BurntSushi

encoding_rs_io

Streaming I/O adapters for the encoding_rs crate.

by BurntSushi

Rust

Rust99.8%
Shell0.2%

Commit activity

Last 12 weeks · 0 commits

Full graph →

Community health

2 of 6 standards met

Community profile →
42
✓README✓License○Contributing○Code of Conduct○Issue Template○PR Template

Recent PRs & issues

Quiet · 3 in progress · Last activity 3mo ago
See all on GitHub →
icecream95

Currently with when operating on many files can be 50x slower without compared to . More than 99% of CPU time is spent in , which is called from , called from , called from here: https://github.com/BurntSushi/ripgrep/blob/master/crates/searcher/src/searcher/mod.rs#L911-L919 If grows large, then the function will clear the entire capacity of the vector for every file, irrespective of the file's size, which in my case resulted in 300 GB of memory transfers for only 3 GB of data. If implemented the function, then it would be able to avoid initializing the entire buffer, only writing to the part of it that actually needs to be written. (Alternatively, could be changed to not call , or to not reuse a single for every file.)

icecream95 · 3y ago
artem-tim

Is there anything in progress/alternative regarding the first point of the Future work paragraph of the README ? An encoder that accepts an arbitrary std::io::Write implementation and takes valid UTF-8 and transcodes it to a selected destination encoding. This encoder would implement std::fmt::Write. If not, would you be open to a PR implementing such encoder ?

artem-tim · 3y ago
this-name-ok

The field in was a that accumulated the total bytes read across the entire stream. On 32-bit targets, this wraps at ~4 GiB, causing to become true again and the BOM logic to incorrectly re-trigger, corrupting the output. Rename to to clarify its actual semantics (position within BOM emission, bounded to 0..=3). Remove the dead accumulation after the BOM phase completes, and set as a sentinel to permanently prevent re-entry into the BOM handling path.

this-name-ok · 3mo ago
BurntSushi

We want to expose the ability to "use Windows-1252, but if there's a BOM, honor the BOM." See @hsivonen's comment here: https://github.com/hsivonen/encoding_rs/issues/8#issuecomment-411356371 I'm not sure if this is something where we can add a new option to the builder for, or if this requires deeper changes in the decoder.

BurntSushi · 4y ago
dralley

I might be missing something, but it seems like contains an internal buffer for transcoded bytes as well as a position in that buffer where reads start from, as well as an end of the transcoded bytes Shouldn't it be possible to trivially implement on top of that without a needing external buffer (and hence, extra copying)?

dralley · 4y ago
Structured data for AI agents

Repository: BurntSushi/encoding_rs_io. Description: Streaming I/O adapters for the encoding_rs crate. Stars: 30, Forks: 10. Primary language: Rust. Languages: Rust (99.8%), Shell (0.2%). Open PRs: 3, open issues: 6. Last activity: 3mo ago. Community health: 42%. Top contributors: BurntSushi, this-name-ok, igor-raits, LesnyRumcajs.