What is Hackers' Pub?

The Death Of Baseload And Similar Grid Tropes

Anyone who has spent any amount of time in or near people who are really interested in energy policies will have heard proclamations such as that ‘baseload is dead’ and the sorting of e…

hackaday.com · Hackaday

Apache Arrow is 10 years old 🎉

The Apache Arrow project was officially established and had its first git commit on February 5th 2016, and we are therefore enthusiastic to announce its 10-year anniversary! Looking back over these 10 years, the project has developed in many unforeseen ways and we believe to have delivered on our objective of providing agnostic, efficient, durable standards for the exchange of columnar data. How it started From the start, Arrow has been a joint effort between practitioners of various horizons looking to build common grounds to efficiently exchange columnar data between different libraries and systems. In this blog post, Julien Le Dem recalls how some of the founders of the Apache Parquet project participated in the early days of the Arrow design phase. The idea of Arrow as an in-memory format was meant to address the other half of the interoperability problem, the natural complement to Parquet as a persistent storage format. Apache Arrow 0.1.0 The first Arrow release, numbered 0.1.0, was tagged on October 7th 2016. It already featured the main data types that are still the bread-and-butter of most Arrow datasets, as evidenced in this Flatbuffers declaration: /// ---------------------------------------------------------------------- /// Top-level Type value, enabling extensible type-specific metadata. We can /// add new logical types to Type without breaking backwards compatibility union Type { Null, Int, FloatingPoint, Binary, Utf8, Bool, Decimal, Date, Time, Timestamp, Interval, List, Struct_, Union } The release announcement made the bold claim that "the metadata and physical data representation should be fairly stable as we have spent time finalizing the details". Does that promise hold? The short answer is: yes, almost! But let us analyse that in a bit more detail: the Columnar format, for the most part, has only seen additions of new datatypes since 2016. One single breaking change occurred: Union types cannot have a top-level validity bitmap anymore. the IPC format has seen several minor evolutions of its framing and metadata format; these evolutions are encoded in the MetadataVersion field which ensures that new readers can read data produced by old writers. The single breaking change is related to the same Union validity change mentioned above. First cross-language integration tests Arrow 0.1.0 had two implementations: C++ and Java, with bindings of the former to Python. There were also no integration tests to speak of, that is, no automated assessment that the two implementations were in sync (what could go wrong?). Integration tests had to wait for November 2016 to be designed, and the first automated CI run probably occurred in December of the same year. Its results cannot be fetched anymore, so we can only assume the tests passed successfully. 🙂 From that moment, integration tests have grown to follow additions to the Arrow format, while ensuring that older data can still be read successfully. For example, the integration tests that are routinely checked against multiple implementations of Arrow have data files generated in 2019 by Arrow 0.14.1. No breaking changes... almost As mentioned above, at some point the Union type lost its top-level validity bitmap, breaking compatibility for the workloads that made use of this feature. This change was proposed back in June 2020 and enacted shortly thereafter. It elicited no controversy and doesn't seem to have caused any significant discontent among users, signaling that the feature was probably not widely used (if at all). Since then, there has been precisely zero breaking change in the Arrow Columnar and IPC formats. Apache Arrow 1.0.0 We have been extremely cautious with version numbering and waited until July 2020 before finally switching away from 0.x version numbers. This was signalling to the world that Arrow had reached its "adult phase" of making formal compatibility promises, and that the Arrow formats were ready for wide consumption amongst the data ecosystem. Apache Arrow, today Describing the breadth of the Arrow ecosystem today would take a full-fledged article of its own, or perhaps even multiple Wikipedia pages. Our "powered by" page can give a small taste. As for the Arrow project, we will merely refer you to our official documentation: The various specifications that cater to multiple aspects of sharing Arrow data, such as in-process zero-copy sharing between producers and consumers that know nothing about each other, or executing database queries that efficiently return their results in the Arrow format. The implementation status page that lists the implementations developed officially under the Apache Arrow umbrella (native software libraries for C, C++, C#, Go, Java, JavaScript, Julia, MATLAB, Python, R, Ruby, and Rust). But keep in mind that multiple third-party implementations exist in non-Apache projects, either open source or proprietary. However, that is only a small part of the landscape. The Arrow project hosts several official subprojects, such as ADBC and nanoarrow. A notable success story is Apache DataFusion, which began as an Arrow subproject and later graduated to become an independent top-level project in the Apache Software Foundation, reflecting the maturity and impact of the technology. Beyond these subprojects, many third-party efforts have adopted the Arrow formats for efficient interoperability. GeoArrow is an impressive example of how building on top of existing Arrow formats and implementations can enable groundbreaking efficiency improvements in a very non-trivial problem space. It should also be noted that Arrow, as an in-memory columnar format, is often used hand in hand with Parquet for persistent storage; as a matter of fact, most official Parquet implementations are nowadays being developed within Arrow repositories (C++, Rust, Go). Tomorrow The Apache Arrow community is primarily driven by consensus, and the project does not have a formal roadmap. We will continue to welcome everyone who wishes to participate constructively. While the specifications are stable, they still welcome additions to cater for new use cases, as they have done in the past. The Arrow implementations are actively maintained, gaining new features, bug fixes, and performance improvements. We encourage people to contribute to their implementation of choice, and to engage with us and the community. Now and going forward, a large amount of Arrow-related progress is happening in the broader ecosystem of third-party tools and libraries. It is no longer possible for us to keep track of all the work being done in those areas, but we are proud to see that they are building on the same stable foundations that have been laid 10 years ago.

arrow.apache.org · Apache Arrow

Analysis: China’s CO2 emissions have now been ‘flat or falling’ for 21 months - Carbon Brief

China’s carbon dioxide (CO2) emissions fell by 1% in the final quarter of 2025, likely securing a decline of 0.3% for the full year as a whole.

www.carbonbrief.org · Carbon Brief

How Young Men View Trump 2.0 After One Year

Third Way is a national think tank and advocacy organization that champions moderate policy and political ideas. Our work on the center left acts as a critical bulwark against political extremism.

www.thirdway.org

An AI Agent Published a Hit Piece on Me

Scott Shambaugh helps maintain the excellent and venerable matplotlib Python charting library, including taking on the thankless task of triaging and reviewing incoming pull requests. A GitHub account called @crabby-rathbun …

simonwillison.net · Simon Willison’s Weblog

차준환이 옳았다… "이보다 잘 탈 수 없어" 외신도 분노한 점수 | 중앙일보

차준환은 10일(한국시간) 이탈리아 밀라노 아이스 스케이팅 아레나에서 열린 2026 밀라노·코르티나담페초 동계올림픽 피겨스케이팅 남자 싱글 쇼트프로그램에서 기술점수(TES) 50.08점, 예술점수(PCS) 42.64점을 받아 총점 92.72점을 기록, 6위에 올랐다. 1위는 108.16점을 받은 일리야 말리닌(미국), 2위는 가기야마 유마(일본), 3위는 프랑스 아담 샤오 힝파(프랑스)가 차지했다. 42.64점은 너무 적다"고 지적했다.

www.joongang.co.kr · 중앙일보

ai;dr | Sid's Blog

www.0xsid.com

TikTok is tracking you, even if you don't use the app. Here's how to stop it

TikTok is growing its data harvesting empire, and avoiding the app won’t protect you – but some easy steps can keep you safe.

www.bbc.com

China has planted so many trees around the Taklamakan Desert that it's turned this 'biological void' into a carbon sink

Huge-scale ecological engineering around the edges of one of the world's largest and driest deserts has turned it into a carbon sink that absorbs more CO2 than it emits, research suggests.

www.livescience.com · Live Science

Pebble Emulator

ericmigi.github.io

Arabic translation contains a dead link to joinfirefish.org · Issue #171 · jointhefediverse-net/jointhefediverse.net

The link in here is dead: jointhefediverse.net/translations/ar/join.json Line 8 in 6c0f9a7 "outro": "يمكنك أيضًا تصفح مجتمعات الفديفيرس استنادًا إلى <mark> الاهتمامات </ mark> على <a href='https://...

github.com · GitHub

ai;dr | Sid's Blog

www.0xsid.com