Here is an example to open a modal dialog page using JavaScript in Oracle Apex.

This example will call a modal dialog page (page number 27) from a normal page (page number 25). Page 25 having a page item P25_EMP_ID, which we will pass to Page 27's item P27_EMP_ID.

JavaScript Code to Open a Modal Dialog Page

Create a dynamic action on the button's click event or any other object. Create a True action to execute the JavaScript code and add the following code in it:

var x = apex.item('P25_EMP_ID').getValue();
var url = "f?p=#APP_ID#:27:#SESSION#::NO:RP,27:P27_EMP_ID:#P25_EMP_ID#";
url = url.replace("#APP_ID#", $v("pFlowId"));
url = url.replace("#SESSION#", $v("pInstance"));
url = url.replace("#P25_EMP_ID#", x);
apex.server.process("PREPARE_URL", {
x01: url
  }, {
  success: function(pData) {
   if (pData.success === true) {
     apex.navigation.redirect(pData.url);
   } else {
     console.log("FALSE");
   }
 },
error: function(request, status, error) {
console.log("status---" + status + " error----" + error);
  }
});

In the above code, change the items and values as explained below:

1. Line-1: var x will store the value of the source page's P25_EMP_ID value. Change it with your source page item name.
2. Line-2: URL variable is creating the Oracle Apex page URL for the modal dialog. Change 27 values in URL to your modal dialog page number. Change the P27_EMP_ID item with your modal dialog page item. Change P25_EMP_ID with your source page item.
3. Apex server process name is PREPARE_URL; use this same name to create the Ajax process as explained below.

Create an Ajax Callback Process

Now click on the process tab and create a new Ajax callback process named PREPARE_URL and add the following PL/SQL code in it:

declare
   result varchar2(32767);
begin
   result:=apex_util.prepare_url(apex_application.g_x01);
   apex_json.open_object;
   apex_json.write('success', true);
   apex_json.write('url', result);
   apex_json.close_object;
exception
 when others then
   apex_json.open_object;
   apex_json.write('success', false);
   apex_json.write('message', sqlerrm);
   apex_json.close_object;
end;

It is done. Now when you click on the button, it will open the modal dialog page you have set.

Reference: Question: Dynamic Action Redirect To Another Page in Oracle Apex