When using immediate mode rendering, you'd fill up a CPU side buffer with primitives, and then send it to the GPU, and then do a draw call. The basic limitation is that you cannot really buffer heterogeneous vertex types (such as some with a vertex attribute enabled, and some with a fallback value, or similar variations, maybe you want some field to be compressed most of the time, etc.). If you want to do so, you would have to set up the entire vertex array descriptor again, basically, and would have to do so for every switch in mode. Obviously you also need to draw all the previously accumulated geometry when you want to switch shaders temporarily for some geometry.
So I was thinking, it makes little sense to treat a vertex buffer as some opaque data storage that then gains meaning through a vertex array's attribute pointers. It effectively can't hold heterogeneous data anyway. So what you would need in practice is to have one vertex array per vertex type (including whether attributes are enabled or disabled, or when using different compression formats), and each is bound to a different buffer, with each buffer basically having a fixed data format. And then you produce immediate-mode data in multiple buffers, send it to the GPU once per buffer, and then replay all the draw calls, switching between the vertex arrays and shaders to maintain the right draw order (in case you are not using the z-Buffer for ordering).
So, for my OpenGL dialect, I would make it so that it's not the vertex array owning the attribute pointers in the form (layout, buffer), but rather, the buffer owns a data layout (just the data structure description, without any reference to a shader's attribute location), and the vertex array would simply select (buffer, fieldID) for each attribute. This way, all geometry data you send into a buffer already has a type description when you send it, which also allows the driver to take care of any alignment requirement quirks the hardware might have (like iOS devices wanting all vertex attributes to be 4-byte aligned, even when the attribute itself is a byte or byte vector). Previously, the driver would be forced to store the bytes you send it, and then create a transformed view on demand when you actually point a vertex array to it and start drawing.
Additionally, since you basically can't mix primitives anyway per buffer, as you need one draw call per primitive type, you could also specify whether a buffer is intended for triangles or lines, etc.. This would also allow you to more easily perform stuff like wireframe rendering even without direct HW support, because the driver would just have to maintain an index buffer that turns e.g. triangles into a wireframe, or one for triangle strips, etc. (assuming a non-indexed draw call here). So I'm proposing for my dialect to have (data, layout, primitive type) be what a buffer holds, and then (buffer, fieldID) be a single vertex attribute pointer, and a vertex array then consists of multiple attribute pointers, or constant fallback values. This shifts all the data layout burden onto the buffers that actually store the data, and the vertex arrays are really only responsible for pointing to data, and when you make a vertex array point to a new buffer, it also immediately adapts to that buffer's layout, theoretically reducing the number of reconfigurations you would need. A vertex array is then also only allowed to point to buffers of the same primitive type. When you reconfigure a buffer's layout descriptor or primitive type, its contents become invalid and need to be rewritten, which means drivers do not have to keep the original copy around for when they need to do theiry own quirky copy of the data for a specific layout).
And then you could also re-introduce a quad mode on the API level (as a user space extension, basically), which can be emulated using a fixed index buffer under the hood. Quads aren't really used in 3D, but especially common in immediate-mode 2D rendering and stuff like font rendering, etc..
Index buffers as exposed to the user should also specify a primitive, and only buffers that are configured as point primitives can be used for indexed drawing. This restricts the usage of buffers, but creates strong guarantees that make the driver's life easier, and also makes debugging / error reporting easier. And as long as you never reconfigure a buffer, and only rewrite its data, you don't really pay much validation cost for these things.
Of course once you start writing to a vertex buffer from a compute shader, things change a bit, because you then you basically have to guarantee there won't be any HW quirks like alignment requirements beyond natural alignment, etc.. But theoretically, the compute shader could use a similar method to what a fragment's pixel output already does, where it adapts to the data type & layout of the render target.
Anyway, this got more rambly than I thought it would, just wanted to share some progress on the GL dialect I'm cooking up, and ask for feedback. I wonder whether I missed anything? Another thing that would be useful actually is for bindless textures, when you have a texture ID in the vertex data, and if the layout descriptor scheme has a special type for that, the driver could translate that into an internal texture handle at upload time, rather than the vertex/fragment shader having to do that.
