从Google网页中通过正则表达式获取json如何转换unicode对象

风之旋 · 发表于 2024-4-13 23:34:01

场景：Google爬虫，获取下拉框搜索关键词，需要获取页面的里面的json字符串如下面：
'{\\x22aa\\x22:{},\\x22abd\\x22:{\\x22abd\\x22:false,\\x22deb\\x22:false,\\x22det\\x22:false},\\x22async\\x22:{},\\x22attn\\x22:{},\\x22bgd\\x22:{\\x22ac\\x22:true,\\x22as\\x22:true,\\x22at\\x22:0,\\x22ea\\x22:true,\\x22ed\\x22:0,\\x22ei\\x22:true,\\x22el\\x22:true,\\x22ep\\x22:true,\\x22er\\x22:true,\\x22et\\x22:0,\\x22eu\\x22:false,\\x22li\\x22:false,\\x22wl\\x22:false},\\x22cdos\\x22:{\\x22cdobsel\\x22:false},\\x22csi\\x22:{},\\x22d\\x22:{},\\x22foot\\x22:{\\x22dv\\x22:\\x22A13OmSDjAawaEKMYZTqdNnEqnlN77Rg\\x22},\\x22gf\\x22:{\\x22pid\\x22:196},\\x22gwc\\x22:{},\\x22hsm\\x22:{},\\x22jsa\\x22:{\\x22csi\\x22:true,\\x22csir\\x22:100},\\x22kyn\\x22:{},\\x22lli\\x22:{},\\x22mb4ZUb\\x22:{},\\x22pHXghd\\x22:{},\\x22sb_wiz\\x22:{\\x22onf\\x22:\\x22EAEYj6vGoaypudTUARjAp8368N6Oq2sY1P78svbOpoFWGIeIjfvlpdGDVhjtkpzIgLGFiJsBMvcBChcKFeibi-eZvei0qOe7hOWtpui0qOiwsQodChvom4vnmb3otKjnu4TlrabmlbDmja7liIbmnpAKGgoY6JuL55m96LSo6LCx57uT5p6c5YiG5p6QCh0KG-ibi-eZvei0qOe7hOWtpuWIhuaekOa1geeoiwoUChLom4vnmb3otKjnu4TlraZwcHQKFwoV6JuL55m96LSo57uE5a2m5YWs5Y-4ChEKD-ibi-eZvee7hOa1i-W6jwoaChjom4vnmb3otKjnu4TlrabmlbDmja7lupMKEgoQ6JuL55m96LSoIOeglOeptgoOCgzotKjosLHmo4DmtYsQRzJpCh0KG-ibi-eZvei0qOacieWTquS6m-enjeexu--8nwodChvom4vnmb3otKjmnInku4DkuYjlhYPntKDvvJ8KJgok6JuL55m96LSo55qE57uE5oiQ6YOo5YiG5piv5LuA5LmI77yfEOQC\\x22,\\x22scq\\x22:\\x22\\x22,\\x22stok\\x22:\\x22DB6NHG6owl0pg5mSRV3RZ-SroY0\\x22,\\x22zps\\x22:\\x22GI-rxqGsqbnU1AEYwKfN-vDejqtrGNT-_LL2zqaBVhiHiI375aXRg1YY7ZKcyICxhYibAQ\\x22},\\x22sf\\x22:{},\\x22sonic\\x22:{},\\x22spch\\x22:{\\x22ae\\x22:\\x22\\\\u8bf7\\\\u68c0\\\\u67e5\\\\u60a8\\\\u7684\\\\u9ea6\\\\u514b\\\\u98ce\\\\u3002\\\\u003Ca href\\x3d\\\\\\x22https://support.google.com/chrome/?p\\x3dui_voice_search\\\\\\x22 target\\x3d\\\\\\x22_blank\\\\\\x22\\\\u003E\\\\u4e86\\\\u89e3\\\\u8be6\\\\u60c5\\\\u003C/a\\\\u003E\\x22,\\x22ak\\x22:\\x22\\x22,\\x22ao\\x22:true,\\x22cd\\x22:0,\\x22fp\\x22:false,\\x22hl\\x22:\\x22zh-Hans-SG\\x22,\\x22im\\x22:\\x22\\\\u70b9\\\\u51fb\\\\u003Cb\\\\u003E\\\\u5141\\\\u8bb8\\\\u003C/b\\\\u003E\\\\u5f00\\\\u59cb\\\\u8bed\\\\u97f3\\\\u641c\\\\u7d22\\x22,\\x22iw\\x22:\\x22\\\\u6b63\\\\u5728\\\\u7b49\\\\u5f85...\\x22,\\x22lm\\x22:\\x22\\\\u6b63\\\\u5728\\\\u6536\\\\u542c\\\\u2026\\x22,\\x22lu\\x22:\\x22\\\\u76ee\\\\u524d\\\\u5c1a\\\\u672a\\\\u63d0\\\\u4f9b%1$s\\\\u7248\\\\u8bed\\\\u97f3\\\\u641c\\\\u7d22\\\\u670d\\\\u52a1\\x22,\\x22mb\\x22:false,\\x22mc\\x22:false,\\x22ne\\x22:\\x22\\\\u672a\\\\u8fde\\\\u63a5\\\\u5230\\\\u4e92\\\\u8054\\\\u7f51\\x22,\\x22nt\\x22:\\x22\\\\u4e0d\\\\u786e\\\\u5b9a\\\\u6240\\\\u8bf4\\\\u5185\\\\u5bb9\\\\uff0c\\\\u003Cspan\\\\u003E\\\\u8bf7\\\\u91cd\\\\u8bd5\\\\u003C/span\\\\u003E\\x22,\\x22nv\\x22:\\x22\\\\u8bf7\\\\u68c0\\\\u67e5\\\\u60a8\\\\u7684\\\\u9ea6\\\\u514b\\\\u98ce\\\\u548c\\\\u97f3\\\\u9891\\\\u7535\\\\u5e73\\\\u3002\\\\u003Ca href\\x3d\\\\\\x22https://support.google.com/chrome/?p\\x3dui_voice_search\\\\\\x22 target\\x3d\\\\\\x22_blank\\\\\\x22\\\\u003E\\\\u4e86\\\\u89e3\\\\u8be6\\\\u60c5\\\\u003C/a\\\\u003E\\x22,\\x22pe\\x22:\\x22\\\\u8bed\\\\u97f3\\\\u641c\\\\u7d22\\\\u529f\\\\u80fd\\\\u5904\\\\u4e8e\\\\u5173\\\\u95ed\\\\u72b6\\\\u6001\\\\u3002\\\\u003Ca href\\x3d\\\\\\x22https://support.google.com/chrome/?p\\x3dui_voice_search\\\\\\x22 target\\x3d\\\\\\x22_blank\\\\\\x22\\\\u003E\\\\u8be6\\\\u7ec6\\\\u4fe1\\\\u606f\\\\u003C/a\\\\u003E\\x22,\\x22ri\\x22:false,\\x22rm\\x22:\\x22\\\\u8bf7\\\\u5f00\\\\u59cb\\\\u8bf4\\\\u8bdd\\x22,\\x22s3\\x22:false,\\x22sa\\x22:false},\\x22tl\\x22:{\\x22rvkey\\x22:\\x22AIzaSyC_9Rt88UMjzgg5pIVArnfuIVkJx4zCdTY\\x22}}'
然后我通过正在表达式获取后，通过json.loads()报一下错误
报错：json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

转换代码如下：

pmc_json = '''
{\\x22aa\\x22:{},\\x22abd\\x22:{\\x22abd\\x22:false,\\x22deb\\x22:false,\\x22det\\x22:false},\\x22async\\x22:{},\\x22attn\\x22:{},\\x22bgd\\x22:{\\x22ac\\x22:true,\\x22as\\x22:true,\\x22at\\x22:0,\\x22ea\\x22:true,\\x22ed\\x22:0,\\x22ei\\x22:true,\\x22el\\x22:true,\\x22ep\\x22:true,\\x22er\\x22:true,\\x22et\\x22:0,\\x22eu\\x22:false,\\x22li\\x22:false,\\x22wl\\x22:false},\\x22cdos\\x22:{\\x22cdobsel\\x22:false},\\x22csi\\x22:{},\\x22d\\x22:{},\\x22foot\\x22:{\\x22dv\\x22:\\x22A13OmSDjAawaEKMYZTqdNnEqnlN77Rg\\x22},\\x22gf\\x22:{\\x22pid\\x22:196},\\x22gwc\\x22:{},\\x22hsm\\x22:{},\\x22jsa\\x22:{\\x22csi\\x22:true,\\x22csir\\x22:100},\\x22kyn\\x22:{},\\x22lli\\x22:{},\\x22mb4ZUb\\x22:{},\\x22pHXghd\\x22:{},\\x22sb_wiz\\x22:{\\x22onf\\x22:\\x22EAEYj6vGoaypudTUARjAp8368N6Oq2sY1P78svbOpoFWGIeIjfvlpdGDVhjtkpzIgLGFiJsBMvcBChcKFeibi-eZvei0qOe7hOWtpui0qOiwsQodChvom4vnmb3otKjnu4TlrabmlbDmja7liIbmnpAKGgoY6JuL55m96LSo6LCx57uT5p6c5YiG5p6QCh0KG-ibi-eZvei0qOe7hOWtpuWIhuaekOa1geeoiwoUChLom4vnmb3otKjnu4TlraZwcHQKFwoV6JuL55m96LSo57uE5a2m5YWs5Y-4ChEKD-ibi-eZvee7hOa1i-W6jwoaChjom4vnmb3otKjnu4TlrabmlbDmja7lupMKEgoQ6JuL55m96LSoIOeglOeptgoOCgzotKjosLHmo4DmtYsQRzJpCh0KG-ibi-eZvei0qOacieWTquS6m-enjeexu--8nwodChvom4vnmb3otKjmnInku4DkuYjlhYPntKDvvJ8KJgok6JuL55m96LSo55qE57uE5oiQ6YOo5YiG5piv5LuA5LmI77yfEOQC\\x22,\\x22scq\\x22:\\x22\\x22,\\x22stok\\x22:\\x22DB6NHG6owl0pg5mSRV3RZ-SroY0\\x22,\\x22zps\\x22:\\x22GI-rxqGsqbnU1AEYwKfN-vDejqtrGNT-_LL2zqaBVhiHiI375aXRg1YY7ZKcyICxhYibAQ\\x22},\\x22sf\\x22:{},\\x22sonic\\x22:{},\\x22spch\\x22:{\\x22ae\\x22:\\x22\\\\u8bf7\\\\u68c0\\\\u67e5\\\\u60a8\\\\u7684\\\\u9ea6\\\\u514b\\\\u98ce\\\\u3002\\\\u003Ca href\\x3d\\\\\\x22https://support.google.com/chrome/?p\\x3dui_voice_search\\\\\\x22 target\\x3d\\\\\\x22_blank\\\\\\x22\\\\u003E\\\\u4e86\\\\u89e3\\\\u8be6\\\\u60c5\\\\u003C/a\\\\u003E\\x22,\\x22ak\\x22:\\x22\\x22,\\x22ao\\x22:true,\\x22cd\\x22:0,\\x22fp\\x22:false,\\x22hl\\x22:\\x22zh-Hans-SG\\x22,\\x22im\\x22:\\x22\\\\u70b9\\\\u51fb\\\\u003Cb\\\\u003E\\\\u5141\\\\u8bb8\\\\u003C/b\\\\u003E\\\\u5f00\\\\u59cb\\\\u8bed\\\\u97f3\\\\u641c\\\\u7d22\\x22,\\x22iw\\x22:\\x22\\\\u6b63\\\\u5728\\\\u7b49\\\\u5f85...\\x22,\\x22lm\\x22:\\x22\\\\u6b63\\\\u5728\\\\u6536\\\\u542c\\\\u2026\\x22,\\x22lu\\x22:\\x22\\\\u76ee\\\\u524d\\\\u5c1a\\\\u672a\\\\u63d0\\\\u4f9b%1$s\\\\u7248\\\\u8bed\\\\u97f3\\\\u641c\\\\u7d22\\\\u670d\\\\u52a1\\x22,\\x22mb\\x22:false,\\x22mc\\x22:false,\\x22ne\\x22:\\x22\\\\u672a\\\\u8fde\\\\u63a5\\\\u5230\\\\u4e92\\\\u8054\\\\u7f51\\x22,\\x22nt\\x22:\\x22\\\\u4e0d\\\\u786e\\\\u5b9a\\\\u6240\\\\u8bf4\\\\u5185\\\\u5bb9\\\\uff0c\\\\u003Cspan\\\\u003E\\\\u8bf7\\\\u91cd\\\\u8bd5\\\\u003C/span\\\\u003E\\x22,\\x22nv\\x22:\\x22\\\\u8bf7\\\\u68c0\\\\u67e5\\\\u60a8\\\\u7684\\\\u9ea6\\\\u514b\\\\u98ce\\\\u548c\\\\u97f3\\\\u9891\\\\u7535\\\\u5e73\\\\u3002\\\\u003Ca href\\x3d\\\\\\x22https://support.google.com/chrome/?p\\x3dui_voice_search\\\\\\x22 target\\x3d\\\\\\x22_blank\\\\\\x22\\\\u003E\\\\u4e86\\\\u89e3\\\\u8be6\\\\u60c5\\\\u003C/a\\\\u003E\\x22,\\x22pe\\x22:\\x22\\\\u8bed\\\\u97f3\\\\u641c\\\\u7d22\\\\u529f\\\\u80fd\\\\u5904\\\\u4e8e\\\\u5173\\\\u95ed\\\\u72b6\\\\u6001\\\\u3002\\\\u003Ca href\\x3d\\\\\\x22https://support.google.com/chrome/?p\\x3dui_voice_search\\\\\\x22 target\\x3d\\\\\\x22_blank\\\\\\x22\\\\u003E\\\\u8be6\\\\u7ec6\\\\u4fe1\\\\u606f\\\\u003C/a\\\\u003E\\x22,\\x22ri\\x22:false,\\x22rm\\x22:\\x22\\\\u8bf7\\\\u5f00\\\\u59cb\\\\u8bf4\\\\u8bdd\\x22,\\x22s3\\x22:false,\\x22sa\\x22:false},\\x22tl\\x22:{\\x22rvkey\\x22:\\x22AIzaSyC_9Rt88UMjzgg5pIVArnfuIVkJx4zCdTY\\x22}}'
'''
pmc_json = bytes(pmc_json, "utf-8").decode("unicode_escape")
pmc = json.loads(pmc_json)

复制代码

结果如下：

{'aa': {}, 'abd': {'abd': False, 'deb': False, 'det': False}, 'async': {}, 'attn': {}, 'bgd': {'ac': True, 'as': True, 'at': 0, 'ea': True, 'ed': 0, 'ei': True, 'el': True, 'ep': True, 'er': True, 'et': 0, 'eu': False, 'li': False, 'wl': False}, 'cdos': {'cdobsel': False}, 'csi': {}, 'd': {}, 'foot': {'dv': 'A13OmSDjAawaEKMYZTqdNnEqnlN77Rg'}, 'gf': {'pid': 196}, 'gwc': {}, 'hsm': {}, 'jsa': {'csi': True, 'csir': 100}, 'kyn': {}, 'lli': {}, 'mb4ZUb': {}, 'pHXghd': {}, 'sb_wiz': {'onf': 'EAEYj6vGoaypudTUARjAp8368N6Oq2sY1P78svbOpoFWGIeIjfvlpdGDVhjtkpzIgLGFiJsBMvcBChcKFeibi-eZvei0qOe7hOWtpui0qOiwsQodChvom4vnmb3otKjnu4TlrabmlbDmja7liIbmnpAKGgoY6JuL55m96LSo6LCx57uT5p6c5YiG5p6QCh0KG-ibi-eZvei0qOe7hOWtpuWIhuaekOa1geeoiwoUChLom4vnmb3otKjnu4TlraZwcHQKFwoV6JuL55m96LSo57uE5a2m5YWs5Y-4ChEKD-ibi-eZvee7hOa1i-W6jwoaChjom4vnmb3otKjnu4TlrabmlbDmja7lupMKEgoQ6JuL55m96LSoIOeglOeptgoOCgzotKjosLHmo4DmtYsQRzJpCh0KG-ibi-eZvei0qOacieWTquS6m-enjeexu--8nwodChvom4vnmb3otKjmnInku4DkuYjlhYPntKDvvJ8KJgok6JuL55m96LSo55qE57uE5oiQ6YOo5YiG5piv5LuA5LmI77yfEOQC', 'scq': '', 'stok': 'DB6NHG6owl0pg5mSRV3RZ-SroY0', 'zps': 'GI-rxqGsqbnU1AEYwKfN-vDejqtrGNT-_LL2zqaBVhiHiI375aXRg1YY7ZKcyICxhYibAQ'}, 'sf': {}, 'sonic': {}, 'spch': {'ae': '请检查您的麦克风。<a href="https://support.google.com/chrome/?p=ui_voice_search" target="_blank">了解详情</a>', 'ak': '', 'ao': True, 'cd': 0, 'fp': False, 'hl': 'zh-Hans-SG', 'im': '点击<b>允许</b>开始语音搜索', 'iw': '正在等待...', 'lm': '正在收听…', 'lu': '目前尚未提供%1$s版语音搜索服务', 'mb': False, 'mc': False, 'ne': '未连接到互联网', 'nt': '不确定所说内容，<span>请重试</span>', 'nv': '请检查您的麦克风和音频电平。<a href="https://support.google.com/chrome/?p=ui_voice_search" target="_blank">了解详情</a>', 'pe': '语音搜索功能处于关闭状态。<a href="https://support.google.com/chrome/?p=ui_voice_search" target="_blank">详细信息</a>', 'ri': False, 'rm': '请开始说话', 's3': False, 'sa': False}, 'tl': {'rvkey': 'AIzaSyC_9Rt88UMjzgg5pIVArnfuIVkJx4zCdTY'}}

复制代码

感想：
解决这个问题，差不多花了我半天的时间。刚开始通过正则表达式获取的json字符，然后通过print直接打印在python终端，看起来还是比较正常的，看不出来有啥问题；
然后我通过print(repr())打印就能明显的看出区别，然后把它编码成utf-8,然后在转化为unicode完美解决这个问题。

来源:https://www.cnblogs.com/xianqingsong/p/18133454
免责声明：由于采集信息均来自互联网，如果侵犯了您的权益，请联系我们【E-Mail:cb@itdo.tech】我们会及时删除侵权内容，谢谢合作！