How to read csv files with a path or file name containing unicode characters?

[Zapotecatl]

Hi,

I'm making a game in Unreal Engine, and I'm using rapidscv as my database handler with csv files. With the UE filesystem classes I can read files that have unicode characters in the paths or filename. For example:

C:\Steam\私のビデオゲーム\myfile.txt
or
C:\Steam\MyVideoGames\マイファイル.txt

However, with rapidcsv I have not been able to read csv files that contain unicode characters.

Any ideas please?

[d99kris]

Hi @Zapotecatl - I'll try look into this on a windows machine when I get a chance, but based on this Stackoverflow post it seems some special unicode path handling is required on Windows, and rapidscsv currently does not provide that.

As a short-term workaround it could be possible to go via std::ifstream / std::ofstream (I have not tested it myself yet though):

std::ifstream instream(std::filesystem::path(u8"こんにちは"));
rapidcsv::Document doc(instream);
...

[Zapotecatl]

Thanks for you answer!

Essentially that works, specifically in C++17 it is std::filesystem::u8path(u8"こんにちは"), for say static strings where you can apply u8. For example:

	std::string path = u8"D:\\Games\\MoreThanWords\\NexusV1\\ファイル名\\voices.csv";
	std::ifstream instream(std::filesystem::u8path(path));
	rapidcsv::Document voices_table;
	voices_table.Load(instream, rapidcsv::LabelParams(0, -1));

However, the problem is that I have a string that has the path, so I can't apply u8.

// std::string path = FStringToString(FPaths::ProjectDir()); Get path of UE and convert into string, the result is the same like:
std::string path = "D:\\Games\\MoreThanWords\\NexusV1\\ファイル名\\";
path  = path + "voices.csv"
// I can't apply u8
std::ifstream instream(std::filesystem::u8path(path));
rapidcsv::Document voices_table;
voices_table.Load(instream, rapidcsv::LabelParams(0, -1));

The before code dosen't work. I've tried some byte conversions but I haven't been able to do it.

[jschueller]

I had to patch to something like this to use to the wide char apis on windows from an utf8 encoded std::string:

    void ReadCsv()
    {
      std::ifstream stream;
      stream.exceptions(std::ifstream::failbit | std::ifstream::badbit);
#ifdef RAPIDCSV_ENABLE_CXX17
      stream.open(std::filesystem::u8path(mPath), std::ios::binary);
#else
      stream.open(mPath, std::ios::binary);
#endif
      ReadCsv(stream);
    }