python 3.x - How to handle min_itemsize exception in writing to pandas HDFStore -


i using pandas hdfstore store dfs have created data.

store = pd.hdfstore(storename, ...) file in downloaded_files:     try:         gzip.open(file) f:             data = json.loads(f.read())             df = json_normalize(data)                store.append(storekey, df, format='table', append=true)     except typeerror:         pass         #file error 

i have received error:

valueerror: trying store string len [82] in [values_block_2] column column has limit of [72]! consider using min_itemsize preset sizes on these columns 

i found possible set min_itemsize column involved not viable solution not know max length encounter , columns encounter problem.

is there solution automatically catch exception , handle each item occur?

i think can way:

store.append(storekey, df, format='table', append=true, min_itemsize={'long_string_column': 200}) 

basically it's similar following create table sql statement:

create table df(   id     int,   str    varchar(200) ); 

where 200 maximal allowed length str column

the following links might helpful:

https://www.google.com/search?q=pandas+valueerror%3a+trying+to+store+a+string+with+len+in+column+but+min_itemsize&pws=0&gl=us&gws_rd=cr

hdfstore.append(string, dataframe) fails when string column contents longer there

pandas pytable: how specify min_itemsize of elements of multiindex


Comments

Popular posts from this blog

sequelize.js - Sequelize group by with association includes id -

android - Robolectric "INTERNET permission is required" -

java - Android raising EPERM (Operation not permitted) when attempting to send UDP packet after network connection -