Commit activity
Last 12 weeks · 1 commit
Community health
2 of 6 standards met
42
✓README✓License○Contributing○Code of Conduct○Issue Template○PR Template
Recent PRs & issues
Active · 3 in progress · Last activity 1mo ago
See all on GitHub →
Currently with when operating on many files can be 50x slower without compared to . More than 99% of CPU time is spent in , which is called from , called from , called from here: https://github.com/BurntSushi/ripgrep/blob/master/crates/searcher/src/searcher/mod.rs#L911-L919 If grows large, then the function will clear the entire capacity of the vector for every file, irrespective of the file's size, which in my case resulted in 300 GB of memory transfers for only 3 GB of data. If implemented the function, then it would be able to avoid initializing the entire buffer, only writing to the part of it that actually needs to be written. (Alternatively, could be changed to not call , or to not reuse a single for every file.)
Is there anything in progress/alternative regarding the first point of the Future work paragraph of the README ? An encoder that accepts an arbitrary std::io::Write implementation and takes valid UTF-8 and transcodes it to a selected destination encoding. This encoder would implement std::fmt::Write. If not, would you be open to a PR implementing such encoder ?
Recent fixes
The field in was a that accumulated the total bytes read across the entire stream. On 32-bit targets, this wraps at ~4 GiB, causing to become true again and the BOM logic to incorrectly re-trigger, corrupting the output. Rename to to clarify its actual semantics (position within BOM emission, bounded to 0..=3). Remove the dead accumulation after the BOM phase completes, and set as a sentinel to permanently prevent re-entry into the BOM handling path.
We want to expose the ability to "use Windows-1252, but if there's a BOM, honor the BOM." See @hsivonen's comment here: https://github.com/hsivonen/encoding_rs/issues/8#issuecomment-411356371 I'm not sure if this is something where we can add a new option to the builder for, or if this requires deeper changes in the decoder.
I might be missing something, but it seems like contains an internal buffer for transcoded bytes as well as a position in that buffer where reads start from, as well as an end of the transcoded bytes Shouldn't it be possible to trivially implement on top of that without a needing external buffer (and hence, extra copying)?
Structured data for AI agents
Repository: BurntSushi/encoding_rs_io. Description: Streaming I/O adapters for the encoding_rs crate. Stars: 30, Forks: 10. Primary language: Rust. Languages: Rust (99.8%), Shell (0.2%). Open PRs: 3, open issues: 6. Last activity: 1mo ago. Community health: 42%. Top contributors: BurntSushi, this-name-ok, igor-raits, LesnyRumcajs.