kandi background
kandi background
Explore Kits
kandi background
Explore Kits
kandi background
Explore Kits
kandi background
Explore Kits
Explore all Bilibili open source software, libraries, packages, source code, cloud functions and APIs.

Popular New Releases in Bilibili

v2.1.7

四月更新 Pro Max

Bilibili Helper v1.2.30

BILIBILI-HELPER-v1.0.3

Bilibili-Evolved

v2.1.7

DisableWinTracking

Bili.Uwp

四月更新 Pro Max

bilibili-helper-o

Bilibili Helper v1.2.30

BILIBILI-HELPER

BILIBILI-HELPER-v1.0.3

Popular Libraries in Bilibili

Trending New libraries in Bilibili

Top Authors in Bilibili

1

5 Libraries

481

2

5 Libraries

1095

3

5 Libraries

1187

4

4 Libraries

43

5

4 Libraries

1023

6

4 Libraries

99

7

4 Libraries

236

8

4 Libraries

493

9

4 Libraries

154

10

3 Libraries

145

1

5 Libraries

481

2

5 Libraries

1095

3

5 Libraries

1187

4

4 Libraries

43

5

4 Libraries

1023

6

4 Libraries

99

7

4 Libraries

236

8

4 Libraries

493

9

4 Libraries

154

10

3 Libraries

145

Trending Kits in Bilibili

No Trending Kits are available at this moment for Bilibili

Trending Discussions on Bilibili

    How to insert JSON containing escape codes into a JSONB column in PostgreSQL using GORM
    Required item not in soup object - BeautifulSoup Python
    How to get each one of the objects from an array which includes three objects?
    Using wininet to download deflate XML on Windows MSVC, but gets broken data
    Merge one audio file and one image file to create a video with ffmpeg
    Web scraping multiple pages in python and writing it into a csv file
    Why can only download the first episode video on bilibili with youtube-dl?

QUESTION

How to insert JSON containing escape codes into a JSONB column in PostgreSQL using GORM

Asked 2021-Jul-08 at 18:16

I'm trying to store the JSON bytes to PostgreSQL, but there's a problem.

\u0000 cannot be converted to text.

As you can see below, the JSON contains escape sequences such as \u0000, which it seems PostgreSQL is interpreting as unicode characters, not JSON strings.

1err := raws.SaveRawData(data, url)
2// if there is "\u0000" in the bytes
3if err.Error() == "ERROR: unsupported Unicode escape sequence (SQLSTATE 22P05)" {
4    // try to remove \u0000, but not work
5    data = bytes.Trim(data, "\u0000")
6    e := raws.SaveRawData(data, url) // save data again
7    if e != nil {
8        return e // return the same error
9    }
10    return nil
11}
12

Origin API data can be access form Here. There is \u0000 in it:

1err := raws.SaveRawData(data, url)
2// if there is "\u0000" in the bytes
3if err.Error() == "ERROR: unsupported Unicode escape sequence (SQLSTATE 22P05)" {
4    // try to remove \u0000, but not work
5    data = bytes.Trim(data, "\u0000")
6    e := raws.SaveRawData(data, url) // save data again
7    if e != nil {
8        return e // return the same error
9    }
10    return nil
11}
12{
13  "code": 0,
14  "message": "0",
15  "ttl": 1,
16  "data": {
17    "bvid": "BV1jb411C7m3",
18    "aid": 42443484,
19    "videos": 1,
20    "tid": 172,
21    "tname": "手机游戏",
22    "copyright": 1,
23    "pic": "http://i0.hdslb.com/bfs/archive/c76ee4798bf2ba0efc8449bcb3577d508321c6c5.jpg",
24    "title": "冰塔:我连你的大招都敢硬抗,所以告诉我谁才是生物女王?!单s冰塔怒砍档案女王巴德尔,谁,才是生物一姐?(手动滑稽)",
25    "pubdate": 1549100438,
26    "ctime": 1549100438,
27    "desc": "bgm:逮虾户\n今天先水一期冰塔的,明天再水\\u0000绿塔的,后天就可以下红莲啦,计划通嘿嘿嘿(º﹃º )",
28    "desc_v2": [
29      {
30        "raw_text": "bgm:逮虾户\n今天先水一期冰塔的,明天再水\\u0000绿塔的,后天就可以下红莲啦,计划通嘿嘿嘿(º﹃º )",
31        "type": 1,
32        "biz_id": 0
33      }
34    ],
35    "state": 0,
36    "duration": 265,
37    "rights": {
38      "bp": 0,
39      "elec": 0,
40      "download": 1,
41      "movie": 0,
42      "pay": 0,
43      "hd5": 0,
44      "no_reprint": 1,
45      "autoplay": 1,
46      "ugc_pay": 0,
47      "is_cooperation": 0,
48      "ugc_pay_preview": 0,
49      "no_background": 0,
50      "clean_mode": 0,
51      "is_stein_gate": 0
52    },
53    "owner": {
54      "mid": 39699039,
55      "name": "明眸-雅望",
56      "face": "http://i0.hdslb.com/bfs/face/240f74f8706955119575ea6c6cb1d31892f93800.jpg"
57    },
58    "stat": {
59      "aid": 42443484,
60      "view": 1107,
61      "danmaku": 7,
62      "reply": 22,
63      "favorite": 5,
64      "coin": 4,
65      "share": 0,
66      "now_rank": 0,
67      "his_rank": 0,
68      "like": 10,
69      "dislike": 0,
70      "evaluation": "",
71      "argue_msg": ""
72    },
73    "dynamic": "#崩坏3#",
74    "cid": 74479750,
75    "dimension": {
76      "width": 1280,
77      "height": 720,
78      "rotate": 0
79    },
80    "no_cache": false,
81    "pages": [
82      {
83        "cid": 74479750,
84        "page": 1,
85        "from": "vupload",
86        "part": "冰塔:我连你的大招都敢硬抗,所以告诉我谁才是生物女王?!单s冰塔怒砍档案女王巴德尔,谁,才是生物一姐?(手动滑稽)",
87        "duration": 265,
88        "vid": "",
89        "weblink": "",
90        "dimension": {
91          "width": 1280,
92          "height": 720,
93          "rotate": 0
94        }
95      }
96    ],
97    "subtitle": {
98      "allow_submit": false,
99      "list": []
100    },
101    "user_garb": {
102      "url_image_ani_cut": ""
103    }
104  }
105}
106

The struct for save is:

1err := raws.SaveRawData(data, url)
2// if there is "\u0000" in the bytes
3if err.Error() == "ERROR: unsupported Unicode escape sequence (SQLSTATE 22P05)" {
4    // try to remove \u0000, but not work
5    data = bytes.Trim(data, "\u0000")
6    e := raws.SaveRawData(data, url) // save data again
7    if e != nil {
8        return e // return the same error
9    }
10    return nil
11}
12{
13  "code": 0,
14  "message": "0",
15  "ttl": 1,
16  "data": {
17    "bvid": "BV1jb411C7m3",
18    "aid": 42443484,
19    "videos": 1,
20    "tid": 172,
21    "tname": "手机游戏",
22    "copyright": 1,
23    "pic": "http://i0.hdslb.com/bfs/archive/c76ee4798bf2ba0efc8449bcb3577d508321c6c5.jpg",
24    "title": "冰塔:我连你的大招都敢硬抗,所以告诉我谁才是生物女王?!单s冰塔怒砍档案女王巴德尔,谁,才是生物一姐?(手动滑稽)",
25    "pubdate": 1549100438,
26    "ctime": 1549100438,
27    "desc": "bgm:逮虾户\n今天先水一期冰塔的,明天再水\\u0000绿塔的,后天就可以下红莲啦,计划通嘿嘿嘿(º﹃º )",
28    "desc_v2": [
29      {
30        "raw_text": "bgm:逮虾户\n今天先水一期冰塔的,明天再水\\u0000绿塔的,后天就可以下红莲啦,计划通嘿嘿嘿(º﹃º )",
31        "type": 1,
32        "biz_id": 0
33      }
34    ],
35    "state": 0,
36    "duration": 265,
37    "rights": {
38      "bp": 0,
39      "elec": 0,
40      "download": 1,
41      "movie": 0,
42      "pay": 0,
43      "hd5": 0,
44      "no_reprint": 1,
45      "autoplay": 1,
46      "ugc_pay": 0,
47      "is_cooperation": 0,
48      "ugc_pay_preview": 0,
49      "no_background": 0,
50      "clean_mode": 0,
51      "is_stein_gate": 0
52    },
53    "owner": {
54      "mid": 39699039,
55      "name": "明眸-雅望",
56      "face": "http://i0.hdslb.com/bfs/face/240f74f8706955119575ea6c6cb1d31892f93800.jpg"
57    },
58    "stat": {
59      "aid": 42443484,
60      "view": 1107,
61      "danmaku": 7,
62      "reply": 22,
63      "favorite": 5,
64      "coin": 4,
65      "share": 0,
66      "now_rank": 0,
67      "his_rank": 0,
68      "like": 10,
69      "dislike": 0,
70      "evaluation": "",
71      "argue_msg": ""
72    },
73    "dynamic": "#崩坏3#",
74    "cid": 74479750,
75    "dimension": {
76      "width": 1280,
77      "height": 720,
78      "rotate": 0
79    },
80    "no_cache": false,
81    "pages": [
82      {
83        "cid": 74479750,
84        "page": 1,
85        "from": "vupload",
86        "part": "冰塔:我连你的大招都敢硬抗,所以告诉我谁才是生物女王?!单s冰塔怒砍档案女王巴德尔,谁,才是生物一姐?(手动滑稽)",
87        "duration": 265,
88        "vid": "",
89        "weblink": "",
90        "dimension": {
91          "width": 1280,
92          "height": 720,
93          "rotate": 0
94        }
95      }
96    ],
97    "subtitle": {
98      "allow_submit": false,
99      "list": []
100    },
101    "user_garb": {
102      "url_image_ani_cut": ""
103    }
104  }
105}
106type RawJSONData struct {
107    ID        uint64         `gorm:"primarykey" json:"id"`
108    CreatedAt time.Time      `json:"-"`
109    DeletedAt gorm.DeletedAt `json:"-" gorm:"index"`
110    Data      datatypes.JSON `json:"data"`
111    URL       string         `gorm:"index" json:"url"`
112}
113

datatypes.JSON is from gorm.io/datatypes. It seems just is json.RawMessage, it is (extend from?) a []byte.

I use PostgreSQL's JSONB type for storage this data.

Table:

1err := raws.SaveRawData(data, url)
2// if there is "\u0000" in the bytes
3if err.Error() == "ERROR: unsupported Unicode escape sequence (SQLSTATE 22P05)" {
4    // try to remove \u0000, but not work
5    data = bytes.Trim(data, "\u0000")
6    e := raws.SaveRawData(data, url) // save data again
7    if e != nil {
8        return e // return the same error
9    }
10    return nil
11}
12{
13  "code": 0,
14  "message": "0",
15  "ttl": 1,
16  "data": {
17    "bvid": "BV1jb411C7m3",
18    "aid": 42443484,
19    "videos": 1,
20    "tid": 172,
21    "tname": "手机游戏",
22    "copyright": 1,
23    "pic": "http://i0.hdslb.com/bfs/archive/c76ee4798bf2ba0efc8449bcb3577d508321c6c5.jpg",
24    "title": "冰塔:我连你的大招都敢硬抗,所以告诉我谁才是生物女王?!单s冰塔怒砍档案女王巴德尔,谁,才是生物一姐?(手动滑稽)",
25    "pubdate": 1549100438,
26    "ctime": 1549100438,
27    "desc": "bgm:逮虾户\n今天先水一期冰塔的,明天再水\\u0000绿塔的,后天就可以下红莲啦,计划通嘿嘿嘿(º﹃º )",
28    "desc_v2": [
29      {
30        "raw_text": "bgm:逮虾户\n今天先水一期冰塔的,明天再水\\u0000绿塔的,后天就可以下红莲啦,计划通嘿嘿嘿(º﹃º )",
31        "type": 1,
32        "biz_id": 0
33      }
34    ],
35    "state": 0,
36    "duration": 265,
37    "rights": {
38      "bp": 0,
39      "elec": 0,
40      "download": 1,
41      "movie": 0,
42      "pay": 0,
43      "hd5": 0,
44      "no_reprint": 1,
45      "autoplay": 1,
46      "ugc_pay": 0,
47      "is_cooperation": 0,
48      "ugc_pay_preview": 0,
49      "no_background": 0,
50      "clean_mode": 0,
51      "is_stein_gate": 0
52    },
53    "owner": {
54      "mid": 39699039,
55      "name": "明眸-雅望",
56      "face": "http://i0.hdslb.com/bfs/face/240f74f8706955119575ea6c6cb1d31892f93800.jpg"
57    },
58    "stat": {
59      "aid": 42443484,
60      "view": 1107,
61      "danmaku": 7,
62      "reply": 22,
63      "favorite": 5,
64      "coin": 4,
65      "share": 0,
66      "now_rank": 0,
67      "his_rank": 0,
68      "like": 10,
69      "dislike": 0,
70      "evaluation": "",
71      "argue_msg": ""
72    },
73    "dynamic": "#崩坏3#",
74    "cid": 74479750,
75    "dimension": {
76      "width": 1280,
77      "height": 720,
78      "rotate": 0
79    },
80    "no_cache": false,
81    "pages": [
82      {
83        "cid": 74479750,
84        "page": 1,
85        "from": "vupload",
86        "part": "冰塔:我连你的大招都敢硬抗,所以告诉我谁才是生物女王?!单s冰塔怒砍档案女王巴德尔,谁,才是生物一姐?(手动滑稽)",
87        "duration": 265,
88        "vid": "",
89        "weblink": "",
90        "dimension": {
91          "width": 1280,
92          "height": 720,
93          "rotate": 0
94        }
95      }
96    ],
97    "subtitle": {
98      "allow_submit": false,
99      "list": []
100    },
101    "user_garb": {
102      "url_image_ani_cut": ""
103    }
104  }
105}
106type RawJSONData struct {
107    ID        uint64         `gorm:"primarykey" json:"id"`
108    CreatedAt time.Time      `json:"-"`
109    DeletedAt gorm.DeletedAt `json:"-" gorm:"index"`
110    Data      datatypes.JSON `json:"data"`
111    URL       string         `gorm:"index" json:"url"`
112}
113create table raw_json_data
114(
115    id         bigserial not null constraint raw_json_data_pke primary key,
116    created_at timestamp with time zone,
117    deleted_at timestamp with time zone,
118    data       jsonb,
119    url        text
120);
121

ANSWER

Answered 2021-Jul-08 at 16:05

Well, I solved it by:

copy icondownload icon

1err := raws.SaveRawData(data, url)
2// if there is "\u0000" in the bytes
3if err.Error() == "ERROR: unsupported Unicode escape sequence (SQLSTATE 22P05)" {
4    // try to remove \u0000, but not work
5    data = bytes.Trim(data, "\u0000")
6    e := raws.SaveRawData(data, url) // save data again
7    if e != nil {
8        return e // return the same error
9    }
10    return nil
11}
12{
13  "code": 0,
14  "message": "0",
15  "ttl": 1,
16  "data": {
17    "bvid": "BV1jb411C7m3",
18    "aid": 42443484,
19    "videos": 1,
20    "tid": 172,
21    "tname": "手机游戏",
22    "copyright": 1,
23    "pic": "http://i0.hdslb.com/bfs/archive/c76ee4798bf2ba0efc8449bcb3577d508321c6c5.jpg",
24    "title": "冰塔:我连你的大招都敢硬抗,所以告诉我谁才是生物女王?!单s冰塔怒砍档案女王巴德尔,谁,才是生物一姐?(手动滑稽)",
25    "pubdate": 1549100438,
26    "ctime": 1549100438,
27    "desc": "bgm:逮虾户\n今天先水一期冰塔的,明天再水\\u0000绿塔的,后天就可以下红莲啦,计划通嘿嘿嘿(º﹃º )",
28    "desc_v2": [
29      {
30        "raw_text": "bgm:逮虾户\n今天先水一期冰塔的,明天再水\\u0000绿塔的,后天就可以下红莲啦,计划通嘿嘿嘿(º﹃º )",
31        "type": 1,
32        "biz_id": 0
33      }
34    ],
35    "state": 0,
36    "duration": 265,
37    "rights": {
38      "bp": 0,
39      "elec": 0,
40      "download": 1,
41      "movie": 0,
42      "pay": 0,
43      "hd5": 0,
44      "no_reprint": 1,
45      "autoplay": 1,
46      "ugc_pay": 0,
47      "is_cooperation": 0,
48      "ugc_pay_preview": 0,
49      "no_background": 0,
50      "clean_mode": 0,
51      "is_stein_gate": 0
52    },
53    "owner": {
54      "mid": 39699039,
55      "name": "明眸-雅望",
56      "face": "http://i0.hdslb.com/bfs/face/240f74f8706955119575ea6c6cb1d31892f93800.jpg"
57    },
58    "stat": {
59      "aid": 42443484,
60      "view": 1107,
61      "danmaku": 7,
62      "reply": 22,
63      "favorite": 5,
64      "coin": 4,
65      "share": 0,
66      "now_rank": 0,
67      "his_rank": 0,
68      "like": 10,
69      "dislike": 0,
70      "evaluation": "",
71      "argue_msg": ""
72    },
73    "dynamic": "#崩坏3#",
74    "cid": 74479750,
75    "dimension": {
76      "width": 1280,
77      "height": 720,
78      "rotate": 0
79    },
80    "no_cache": false,
81    "pages": [
82      {
83        "cid": 74479750,
84        "page": 1,
85        "from": "vupload",
86        "part": "冰塔:我连你的大招都敢硬抗,所以告诉我谁才是生物女王?!单s冰塔怒砍档案女王巴德尔,谁,才是生物一姐?(手动滑稽)",
87        "duration": 265,
88        "vid": "",
89        "weblink": "",
90        "dimension": {
91          "width": 1280,
92          "height": 720,
93          "rotate": 0
94        }
95      }
96    ],
97    "subtitle": {
98      "allow_submit": false,
99      "list": []
100    },
101    "user_garb": {
102      "url_image_ani_cut": ""
103    }
104  }
105}
106type RawJSONData struct {
107    ID        uint64         `gorm:"primarykey" json:"id"`
108    CreatedAt time.Time      `json:"-"`
109    DeletedAt gorm.DeletedAt `json:"-" gorm:"index"`
110    Data      datatypes.JSON `json:"data"`
111    URL       string         `gorm:"index" json:"url"`
112}
113create table raw_json_data
114(
115    id         bigserial not null constraint raw_json_data_pke primary key,
116    created_at timestamp with time zone,
117    deleted_at timestamp with time zone,
118    data       jsonb,
119    url        text
120);
121str := string(data)
122str = strings.ReplaceAll(str, `\u0000`, "")
123

Source https://stackoverflow.com/questions/68303357

Community Discussions contain sources that include Stack Exchange Network

    How to insert JSON containing escape codes into a JSONB column in PostgreSQL using GORM
    Required item not in soup object - BeautifulSoup Python
    How to get each one of the objects from an array which includes three objects?
    Using wininet to download deflate XML on Windows MSVC, but gets broken data
    Merge one audio file and one image file to create a video with ffmpeg
    Web scraping multiple pages in python and writing it into a csv file
    Why can only download the first episode video on bilibili with youtube-dl?

QUESTION

How to insert JSON containing escape codes into a JSONB column in PostgreSQL using GORM

Asked 2021-Jul-08 at 18:16

I'm trying to store the JSON bytes to PostgreSQL, but there's a problem.

\u0000 cannot be converted to text.

As you can see below, the JSON contains escape sequences such as \u0000, which it seems PostgreSQL is interpreting as unicode characters, not JSON strings.

1err := raws.SaveRawData(data, url)
2// if there is "\u0000" in the bytes
3if err.Error() == "ERROR: unsupported Unicode escape sequence (SQLSTATE 22P05)" {
4    // try to remove \u0000, but not work
5    data = bytes.Trim(data, "\u0000")
6    e := raws.SaveRawData(data, url) // save data again
7    if e != nil {
8        return e // return the same error
9    }
10    return nil
11}
12

Origin API data can be access form Here. There is \u0000 in it:

1err := raws.SaveRawData(data, url)
2// if there is "\u0000" in the bytes
3if err.Error() == "ERROR: unsupported Unicode escape sequence (SQLSTATE 22P05)" {
4    // try to remove \u0000, but not work
5    data = bytes.Trim(data, "\u0000")
6    e := raws.SaveRawData(data, url) // save data again
7    if e != nil {
8        return e // return the same error
9    }
10    return nil
11}
12{
13  "code": 0,
14  "message": "0",
15  "ttl": 1,
16  "data": {
17    "bvid": "BV1jb411C7m3",
18    "aid": 42443484,
19    "videos": 1,
20    "tid": 172,
21    "tname": "手机游戏",
22    "copyright": 1,
23    "pic": "http://i0.hdslb.com/bfs/archive/c76ee4798bf2ba0efc8449bcb3577d508321c6c5.jpg",
24    "title": "冰塔:我连你的大招都敢硬抗,所以告诉我谁才是生物女王?!单s冰塔怒砍档案女王巴德尔,谁,才是生物一姐?(手动滑稽)",
25    "pubdate": 1549100438,
26    "ctime": 1549100438,
27    "desc": "bgm:逮虾户\n今天先水一期冰塔的,明天再水\\u0000绿塔的,后天就可以下红莲啦,计划通嘿嘿嘿(º﹃º )",
28    "desc_v2": [
29      {
30        "raw_text": "bgm:逮虾户\n今天先水一期冰塔的,明天再水\\u0000绿塔的,后天就可以下红莲啦,计划通嘿嘿嘿(º﹃º )",
31        "type": 1,
32        "biz_id": 0
33      }
34    ],
35    "state": 0,
36    "duration": 265,
37    "rights": {
38      "bp": 0,
39      "elec": 0,
40      "download": 1,
41      "movie": 0,
42      "pay": 0,
43      "hd5": 0,
44      "no_reprint": 1,
45      "autoplay": 1,
46      "ugc_pay": 0,
47      "is_cooperation": 0,
48      "ugc_pay_preview": 0,
49      "no_background": 0,
50      "clean_mode": 0,
51      "is_stein_gate": 0
52    },
53    "owner": {
54      "mid": 39699039,
55      "name": "明眸-雅望",
56      "face": "http://i0.hdslb.com/bfs/face/240f74f8706955119575ea6c6cb1d31892f93800.jpg"
57    },
58    "stat": {
59      "aid": 42443484,
60      "view": 1107,
61      "danmaku": 7,
62      "reply": 22,
63      "favorite": 5,
64      "coin": 4,
65      "share": 0,
66      "now_rank": 0,
67      "his_rank": 0,
68      "like": 10,
69      "dislike": 0,
70      "evaluation": "",
71      "argue_msg": ""
72    },
73    "dynamic": "#崩坏3#",
74    "cid": 74479750,
75    "dimension": {
76      "width": 1280,
77      "height": 720,
78      "rotate": 0
79    },
80    "no_cache": false,
81    "pages": [
82      {
83        "cid": 74479750,
84        "page": 1,
85        "from": "vupload",
86        "part": "冰塔:我连你的大招都敢硬抗,所以告诉我谁才是生物女王?!单s冰塔怒砍档案女王巴德尔,谁,才是生物一姐?(手动滑稽)",
87        "duration": 265,
88        "vid": "",
89        "weblink": "",
90        "dimension": {
91          "width": 1280,
92          "height": 720,
93          "rotate": 0
94        }
95      }
96    ],
97    "subtitle": {
98      "allow_submit": false,
99      "list": []
100    },
101    "user_garb": {
102      "url_image_ani_cut": ""
103    }
104  }
105}
106

The struct for save is:

1err := raws.SaveRawData(data, url)
2// if there is "\u0000" in the bytes
3if err.Error() == "ERROR: unsupported Unicode escape sequence (SQLSTATE 22P05)" {
4    // try to remove \u0000, but not work
5    data = bytes.Trim(data, "\u0000")
6    e := raws.SaveRawData(data, url) // save data again
7    if e != nil {
8        return e // return the same error
9    }
10    return nil
11}
12{
13  "code": 0,
14  "message": "0",
15  "ttl": 1,
16  "data": {
17    "bvid": "BV1jb411C7m3",
18    "aid": 42443484,
19    "videos": 1,
20    "tid": 172,
21    "tname": "手机游戏",
22    "copyright": 1,
23    "pic": "http://i0.hdslb.com/bfs/archive/c76ee4798bf2ba0efc8449bcb3577d508321c6c5.jpg",
24    "title": "冰塔:我连你的大招都敢硬抗,所以告诉我谁才是生物女王?!单s冰塔怒砍档案女王巴德尔,谁,才是生物一姐?(手动滑稽)",
25    "pubdate": 1549100438,
26    "ctime": 1549100438,
27    "desc": "bgm:逮虾户\n今天先水一期冰塔的,明天再水\\u0000绿塔的,后天就可以下红莲啦,计划通嘿嘿嘿(º﹃º )",
28    "desc_v2": [
29      {
30        "raw_text": "bgm:逮虾户\n今天先水一期冰塔的,明天再水\\u0000绿塔的,后天就可以下红莲啦,计划通嘿嘿嘿(º﹃º )",
31        "type": 1,
32        "biz_id": 0
33      }
34    ],
35    "state": 0,
36    "duration": 265,
37    "rights": {
38      "bp": 0,
39      "elec": 0,
40      "download": 1,
41      "movie": 0,
42      "pay": 0,
43      "hd5": 0,
44      "no_reprint": 1,
45      "autoplay": 1,
46      "ugc_pay": 0,
47      "is_cooperation": 0,
48      "ugc_pay_preview": 0,
49      "no_background": 0,
50      "clean_mode": 0,
51      "is_stein_gate": 0
52    },
53    "owner": {
54      "mid": 39699039,
55      "name": "明眸-雅望",
56      "face": "http://i0.hdslb.com/bfs/face/240f74f8706955119575ea6c6cb1d31892f93800.jpg"
57    },
58    "stat": {
59      "aid": 42443484,
60      "view": 1107,
61      "danmaku": 7,
62      "reply": 22,
63      "favorite": 5,
64      "coin": 4,
65      "share": 0,
66      "now_rank": 0,
67      "his_rank": 0,
68      "like": 10,
69      "dislike": 0,
70      "evaluation": "",
71      "argue_msg": ""
72    },
73    "dynamic": "#崩坏3#",
74    "cid": 74479750,
75    "dimension": {
76      "width": 1280,
77      "height": 720,
78      "rotate": 0
79    },
80    "no_cache": false,
81    "pages": [
82      {
83        "cid": 74479750,
84        "page": 1,
85        "from": "vupload",
86        "part": "冰塔:我连你的大招都敢硬抗,所以告诉我谁才是生物女王?!单s冰塔怒砍档案女王巴德尔,谁,才是生物一姐?(手动滑稽)",
87        "duration": 265,
88        "vid": "",
89        "weblink": "",
90        "dimension": {
91          "width": 1280,
92          "height": 720,
93          "rotate": 0
94        }
95      }
96    ],
97    "subtitle": {
98      "allow_submit": false,
99      "list": []
100    },
101    "user_garb": {
102      "url_image_ani_cut": ""
103    }
104  }
105}
106type RawJSONData struct {
107    ID        uint64         `gorm:"primarykey" json:"id"`
108    CreatedAt time.Time      `json:"-"`
109    DeletedAt gorm.DeletedAt `json:"-" gorm:"index"`
110    Data      datatypes.JSON `json:"data"`
111    URL       string         `gorm:"index" json:"url"`
112}
113

datatypes.JSON is from gorm.io/datatypes. It seems just is json.RawMessage, it is (extend from?) a []byte.

I use PostgreSQL's JSONB type for storage this data.

Table:

1err := raws.SaveRawData(data, url)
2// if there is "\u0000" in the bytes
3if err.Error() == "ERROR: unsupported Unicode escape sequence (SQLSTATE 22P05)" {
4    // try to remove \u0000, but not work
5    data = bytes.Trim(data, "\u0000")
6    e := raws.SaveRawData(data, url) // save data again
7    if e != nil {
8        return e // return the same error
9    }
10    return nil
11}
12{
13  "code": 0,
14  "message": "0",
15  "ttl": 1,
16  "data": {
17    "bvid": "BV1jb411C7m3",
18    "aid": 42443484,
19    "videos": 1,
20    "tid": 172,
21    "tname": "手机游戏",
22    "copyright": 1,
23    "pic": "http://i0.hdslb.com/bfs/archive/c76ee4798bf2ba0efc8449bcb3577d508321c6c5.jpg",
24    "title": "冰塔:我连你的大招都敢硬抗,所以告诉我谁才是生物女王?!单s冰塔怒砍档案女王巴德尔,谁,才是生物一姐?(手动滑稽)",
25    "pubdate": 1549100438,
26    "ctime": 1549100438,
27    "desc": "bgm:逮虾户\n今天先水一期冰塔的,明天再水\\u0000绿塔的,后天就可以下红莲啦,计划通嘿嘿嘿(º﹃º )",
28    "desc_v2": [
29      {
30        "raw_text": "bgm:逮虾户\n今天先水一期冰塔的,明天再水\\u0000绿塔的,后天就可以下红莲啦,计划通嘿嘿嘿(º﹃º )",
31        "type": 1,
32        "biz_id": 0
33      }
34    ],
35    "state": 0,
36    "duration": 265,
37    "rights": {
38      "bp": 0,
39      "elec": 0,
40      "download": 1,
41      "movie": 0,
42      "pay": 0,
43      "hd5": 0,
44      "no_reprint": 1,
45      "autoplay": 1,
46      "ugc_pay": 0,
47      "is_cooperation": 0,
48      "ugc_pay_preview": 0,
49      "no_background": 0,
50      "clean_mode": 0,
51      "is_stein_gate": 0
52    },
53    "owner": {
54      "mid": 39699039,
55      "name": "明眸-雅望",
56      "face": "http://i0.hdslb.com/bfs/face/240f74f8706955119575ea6c6cb1d31892f93800.jpg"
57    },
58    "stat": {
59      "aid": 42443484,
60      "view": 1107,
61      "danmaku": 7,
62      "reply": 22,
63      "favorite": 5,
64      "coin": 4,
65      "share": 0,
66      "now_rank": 0,
67      "his_rank": 0,
68      "like": 10,
69      "dislike": 0,
70      "evaluation": "",
71      "argue_msg": ""
72    },
73    "dynamic": "#崩坏3#",
74    "cid": 74479750,
75    "dimension": {
76      "width": 1280,
77      "height": 720,
78      "rotate": 0
79    },
80    "no_cache": false,
81    "pages": [
82      {
83        "cid": 74479750,
84        "page": 1,
85        "from": "vupload",
86        "part": "冰塔:我连你的大招都敢硬抗,所以告诉我谁才是生物女王?!单s冰塔怒砍档案女王巴德尔,谁,才是生物一姐?(手动滑稽)",
87        "duration": 265,
88        "vid": "",
89        "weblink": "",
90        "dimension": {
91          "width": 1280,
92          "height": 720,
93          "rotate": 0
94        }
95      }
96    ],
97    "subtitle": {
98      "allow_submit": false,
99      "list": []
100    },
101    "user_garb": {
102      "url_image_ani_cut": ""
103    }
104  }
105}
106type RawJSONData struct {
107    ID        uint64         `gorm:"primarykey" json:"id"`
108    CreatedAt time.Time      `json:"-"`
109    DeletedAt gorm.DeletedAt `json:"-" gorm:"index"`
110    Data      datatypes.JSON `json:"data"`
111    URL       string         `gorm:"index" json:"url"`
112}
113create table raw_json_data
114(
115    id         bigserial not null constraint raw_json_data_pke primary key,
116    created_at timestamp with time zone,
117    deleted_at timestamp with time zone,
118    data       jsonb,
119    url        text
120);
121

ANSWER

Answered 2021-Jul-08 at 16:05

Well, I solved it by:

copy icondownload icon

1err := raws.SaveRawData(data, url)
2// if there is "\u0000" in the bytes
3if err.Error() == "ERROR: unsupported Unicode escape sequence (SQLSTATE 22P05)" {
4    // try to remove \u0000, but not work
5    data = bytes.Trim(data, "\u0000")
6    e := raws.SaveRawData(data, url) // save data again
7    if e != nil {
8        return e // return the same error
9    }
10    return nil
11}
12{
13  "code": 0,
14  "message": "0",
15  "ttl": 1,
16  "data": {
17    "bvid": "BV1jb411C7m3",
18    "aid": 42443484,
19    "videos": 1,
20    "tid": 172,
21    "tname": "手机游戏",
22    "copyright": 1,
23    "pic": "http://i0.hdslb.com/bfs/archive/c76ee4798bf2ba0efc8449bcb3577d508321c6c5.jpg",
24    "title": "冰塔:我连你的大招都敢硬抗,所以告诉我谁才是生物女王?!单s冰塔怒砍档案女王巴德尔,谁,才是生物一姐?(手动滑稽)",
25    "pubdate": 1549100438,
26    "ctime": 1549100438,
27    "desc": "bgm:逮虾户\n今天先水一期冰塔的,明天再水\\u0000绿塔的,后天就可以下红莲啦,计划通嘿嘿嘿(º﹃º )",
28    "desc_v2": [
29      {
30        "raw_text": "bgm:逮虾户\n今天先水一期冰塔的,明天再水\\u0000绿塔的,后天就可以下红莲啦,计划通嘿嘿嘿(º﹃º )",
31        "type": 1,
32        "biz_id": 0
33      }
34    ],
35    "state": 0,
36    "duration": 265,
37    "rights": {
38      "bp": 0,
39      "elec": 0,
40      "download": 1,
41      "movie": 0,
42      "pay": 0,
43      "hd5": 0,
44      "no_reprint": 1,
45      "autoplay": 1,
46      "ugc_pay": 0,
47      "is_cooperation": 0,
48      "ugc_pay_preview": 0,
49      "no_background": 0,
50      "clean_mode": 0,
51      "is_stein_gate": 0
52    },
53    "owner": {
54      "mid": 39699039,
55      "name": "明眸-雅望",
56      "face": "http://i0.hdslb.com/bfs/face/240f74f8706955119575ea6c6cb1d31892f93800.jpg"
57    },
58    "stat": {
59      "aid": 42443484,
60      "view": 1107,
61      "danmaku": 7,
62      "reply": 22,
63      "favorite": 5,
64      "coin": 4,
65      "share": 0,
66      "now_rank": 0,
67      "his_rank": 0,
68      "like": 10,
69      "dislike": 0,
70      "evaluation": "",
71      "argue_msg": ""
72    },
73    "dynamic": "#崩坏3#",
74    "cid": 74479750,
75    "dimension": {
76      "width": 1280,
77      "height": 720,
78      "rotate": 0
79    },
80    "no_cache": false,
81    "pages": [
82      {
83        "cid": 74479750,
84        "page": 1,
85        "from": "vupload",
86        "part": "冰塔:我连你的大招都敢硬抗,所以告诉我谁才是生物女王?!单s冰塔怒砍档案女王巴德尔,谁,才是生物一姐?(手动滑稽)",
87        "duration": 265,
88        "vid": "",
89        "weblink": "",
90        "dimension": {
91          "width": 1280,
92          "height": 720,
93          "rotate": 0
94        }
95      }
96    ],
97    "subtitle": {
98      "allow_submit": false,
99      "list": []
100    },
101    "user_garb": {
102      "url_image_ani_cut": ""
103    }
104  }
105}
106type RawJSONData struct {
107    ID        uint64         `gorm:"primarykey" json:"id"`
108    CreatedAt time.Time      `json:"-"`
109    DeletedAt gorm.DeletedAt `json:"-" gorm:"index"`
110    Data      datatypes.JSON `json:"data"`
111    URL       string         `gorm:"index" json:"url"`
112}
113create table raw_json_data
114(
115    id         bigserial not null constraint raw_json_data_pke primary key,
116    created_at timestamp with time zone,
117    deleted_at timestamp with time zone,
118    data       jsonb,
119    url        text
120);
121str := string(data)
122str = strings.ReplaceAll(str, `\u0000`, "")
123

Source https://stackoverflow.com/questions/68303357