I had similar experiences asking ChatGPT (3.5) to generate simple snippets of SAS code, like to log the current timestamp to a file every 5 minutes. It gave an authoritative answer. At first glance, it looked plausible, but it wouldn't run, even with some tweaking.

As SAS is relatively less popular than Python and JavaScript, it's a lack of the model's experience with SAS, but as the models become larger and more complex with more training data and more model parameters, they will master SAS. Even now, GPT 4 may do much better with SAS, but I haven't paid for access to it.