Lately I have been working on improving my data handling and storage process. The previous set up I used was to export all data into .csv files, and then import it to SAS and from there creating SAS tables from where I could do my analysis.
This was an easy set up, but not very efficient.
So, now I have set up a proper SQL database. I chose the PostgreSQL as its free and supports unlimited size tables. Now my bot speak directly to the SQL database, inserting and retrieving data directly.
So far so good, it seems to be working without any issues - and it seems to keep the tables smaller (in size) than the SAS tables.
The improvement continues on all parts! Good data is essential for good analysis.