PROBLEM:

While enabling archive mode in database, got below error.

SQL> alter database archivelog;
alter database archivelog
*
ERROR at line 1:
ORA-01126: database must be mounted in this instance and not open in any instance

SOLUTION:

The reported error is because changing archive mode need to be done when the database is in mount stage only. If we try to do it in open state, this error will occur.

So to avoid this error, start the database in mount stage and change archive mode.