So I'm trying to add Lua for scripting and I have a rough idea of what I want do do, but trying to actually execute it has been rather challenging. I know that I probably want to access and mutate components through scripts and there's two solutions I thought of. The first solution is to have a ScriptComponent that has the path to the script to be executed this script would be able to access and mutate components of the entity it's attached to. The second solution is to allow scripts to access any entity and their components which I don't think is that bad of an idea.

The challenge/confusion is actually running the scripts I don't think I want to just reexecute the script each frame so would I just run the script once and then use callbacks?

Here's an example of how I'd like to write the lua code I've spent a lot of time with Roblox and LOVE2D so it's kind of based off of that.

local playerEntity = scene.GetEntity("Player")
local playerMoveComponent = playerEntity.GetCommponent("MoveComponent")

function Update(deltaTime)
    if input.isKeyDown("w") then
        playerMoveComponent:SetDirection()
    end
end

-- another option
function keypressed(key) 

end

This is probably a generic question so there's probably a plethora of angles to approach this, but any tips and suggestions to get me on the right track would be appreciated.

For a small game it often isn't worth it, since it's the same programmer doing all the work but now instead of just supporting a single project in one language, you're implementing code in two languages plus the related tooling, additional debugging, and additional processing. Typically you add it for larger projects so you can have less-competent programmers working in a simpler and safer environment of Lua.

In any event…

A common approach is state-based processing. Set the state you're currently in, and every update branch to that state's condition to continue processing. Often that's a giant switch statement, but could just as easily be an if/else tree or calling into a function stored as a variable as the closest equivalent of function pointers or virtual functions, if those are the preferred way to do it.

Lua has coroutines you can use for some work. Basically you call coroutine.yield() and the processing in that code body returns and then resumes again where it left off with coroutine.resume().

As for callbacks and event handlers, you can create those if you want. You can also cross language boundaries with it. You'll need to store the function that is registered, and when the event happens you'll need to see if a function is registered and if so, call it. Crossing language boundaries is a little trickier, but with lua_isfunction(), lua_pcall(), and using the stack to push/pop arguments, you can cause an event in your C++ code to trigger a function in Lua.

In my game the C++ core handles entity movement, but unlike what frob says I over time naturally started shifting a lot of the game logic onto the Lua side, leading me to wonder if I'm doing everything okay. It also brings up questions of how the entity-component system should be extended by the Lua side, because the need is inevitable, and having the Lua side keep a table for custom components seems wrong, esp. considering networking into account.

In-game events are held in a giant array with maximum size, and are referenced with indices within entities and other things, so they can be easily serialized. Whenever a map (Lua script) loads, it sets its events in the array.

Out-game events like key presses or updates are just functions in a global game table.

I did say “often” and “usually”. If it works for you as an individual, go for it.