Quantcast
Channel: Intel Communities: Message List
Viewing all 18600 articles
Browse latest View live

Intel blog post on the basics of stereoscopic vision


Re: Optane 905p file carving digital forensics

$
0
0

You are 100% right. I concluded for myself that HDD RAID5 not as system drive will profit of Optane. Now when I read my response again, I might not have translated it not correctly in English language.

 

The part with NVME system disk is clear. Also about write speed. I conclude again for myself:

If I have a large imaga file on a RAID5 (HDD), let's say 1500GB cutted in 2GB peaces, which need to be carved by forensic software, the optane drive might not help that much because there are no common files. Right?

 

The software itself might be fast enought already, if run on an NVME drive. Only the cache drive, if the software support this, could be an optane. Right?

 

I ask this all, because processing that large files takes many hours or even days. So a speed boost cuts down the time and saves money.

 

I don't want to upset you in any way. Sorry if I have done.

Re: I210 Teaming after Windows 10 April 2018 Update

$
0
0

Hi Ricoc and Morpheus82,

 Thank you for the information, we will further investigate. 

Regards,
Sharon T
Intel Customer Support
Agent under contract to Intel
 

Re: no response yet from ASRock on bios

$
0
0

Tried a few more times after getting some advice from Steve Simmons (a fellow poster). Thought I have everything setup but still unable to enable optane. See attached screenshots, I have BIOS with pre-OS RST 16 driver. All physical disks set to non-raid. Left unallocated space of 50G at end of disk (right hand side in disk manager view?) for both EVO 970 OS disk as well as SATA HDD disk just in case. See attached photo of error message I can (incompatible disk with unknown problem !!). Not sure what else I should try. Seem strange to me both System OS disk and Optane memory are on the same port number 0. Can you help please.

20180721_164527.jpg20180721_164556.jpg20180721_164255.jpg

Re: How to know xeon gold 6126 processor's turbo steps..

$
0
0

Hi Miguel C,

 

Thanks for the reply, as I mentioned in the description; I already have that information which you shared.

I am looking for number of turbo steps as I furnished the table of v4 processor (looking for centre column numbers for 6126).

 

 

Thanks,

Ashok Kumar Pakki

MobileDemand release rugged tablets with RealSense D415 option

$
0
0

Hi everyone,

 

In July 2018, MobileDemand are releasing a rugged Windows tablet, the xTablet T1270, with the RealSense D415 as an option.

 

https://www.ruggedtabletpc.com/products/xtablet-T1270

 

Also coming is the T1150 Intel Atom model with D415.

 

https://www.ruggedtabletpc.com/products/xTablet-T1150

 

Further details about how MobileDemand is using the D415 can be found here:

 

https://www.ruggedtabletpc.com/blog/3d-camera-for-rugged-tablet

 

MobileDemand will also be offering the option of attaching D415 to more of their tablet models, or to their brand of Surface and iPad Pro cases, via a protective boot.

 

https://www.ruggedtabletpc.com/products/rugged-ipad-case

 

*  Please note that the RealSense 400 Series cameras and their RealSense SDK 2.0 software are not currently supported on iPad.

Re: RMM4 module not working

Re: Microcode revision 0x32 for CPUID 506C9?

$
0
0

Scott Pearson wrote:

 

RE: Can safely be assumed that Meltdown is fixed by the microcode update when reported as this, and microcode updated early by Linux?

 

No! The microcode fixes specifically address only Spectre Variants 2 & 3. They DO NOT address Meltdown! Meltdown is addressed by specific workarounds introduced at the O/S level.

 

...S

Meltdown and Spectre Variant 3 mean the same: Rogue Data Cache Load (RDCL) CVE-2017-5754 .

So: Microcode fixes potentially address also Meltdown.

 

With the latest microcode revision 0x32 (Apollo Lake CPUID 506C9) RDCL_NO is set when loaded early by the Linux kernel (but not by BIOS).

The issue is that the kernel disables PTI mitigation for Meltdown then. Therefore the question:

 

@Intel support: Can safely be assumed that Spectre Variant 3 (Meltdown) is fixed when the microcode update 0x32 for CPUID 506C9 is updated early by Linux?


Very poor performance with UHD630 in WoW after latest patch (BFA addon)

$
0
0

So... Blizzard has removed fullscreen mode with the latest World of Warcraft patch (upcoming BFA addon) and goes for DX12. Only windowed mode and "fullscreen" windowed mode are still supported. After this chance, I can't get more than 20-35FPS anymore in WoW on my laptop with UHD630. Before in fullscreen mode,I had flawless 60FPS on low settings and could easily play WoW on my laptop.

 

It seems UHD630 isnt able to use windowed mode properly. DX12 doesnt support fullscreen render mode anymore it seems, though UHD630 is stated as DX12 compatible.

 

So whats the culprit here? Bug in Intel drivers? In Windows 10? I dont think the problem lies in WoW. This is a huge issue and makes playing WoW literally now broken on Intel iGPUs.

 

The laptop has Optimus too with Nvidia GTX1050Ti, and switching to the Nvidia card also shows now a massive performance drop in WoW. It shows both GPUs are being used for whatever reason, Intel GPU usage next to the Nvidia one is also 60%. This change seems to have also impact on Optimus based solutions.

 

Very low FPS due do Fullscreen windowed mode - World of Warcraft Forums

Re: Intel NUC7PJYH Unable to Boot

$
0
0

Hello Ben,

You should ask your question in a new thread. You have not informed what is a P/N of your RAM.

Any way try to recover bios from Power Button Menu:

 

  1. Prepare USB stick fully formatted to FAT32 (disable quick format option during format). Format your USB on Windows machine (rather than Linux or MAC). Save the Bios file JY0041.bio (Download BIOS Update [JYGLKCPX.86A] ) on this stick and insert it into USB slot (NUC shall be OFF).
  2. Press Power Button for about 3 seconds. You should release the Power Button soon as the power led changes color from blue to amber. NUC will reboot into Power Button Menu.
  3. Press F4 and the recovery shall start  (it can take up to 30 seconds for messages to appear on the screen).
  4. When the recovery finishes, pull out the power cord.
  5. Replace the power cord and press Power Button to switch the computer to ON.
  6. Enter Bios setting by pressing F2 during boot process.
  7. Press F9(followed by "Y"), to set Bios to default settings. Press F10 (followed by "Y"), to save the settings and exit to O.S. Let the NUC fully reboot.
  8. You can enter again to Bios setting to change the necessary settings.

Note:
If for some reason you have difficulty to boot into Power Button Menu, you shall recover your Bios using Security Jumper method BIOS Recovery Update Instructions for Intel® NUC

 

Leon

Copy process / connection break down when sending files over 9260 card

$
0
0

I have the following issue with an Intel 9260 wifi card. The moment I send (larger) files into the LAN (for example samba or Windows 10 network share), the copy process of Windows explorer (with a high chance, does not happen every time, but with a very high chance):

 

- goes to 0kb/s then jumps up again after some time or

- Windows gives an internal network error (Wifi driver seems to crash)

 

While this happens, and having a permanent ping open next to the copy, the moment the copy breaks down, the Acces Point isn't reachable anymore:

 

Z8S4OaD.png

The images show, that after 1-2 seconds of the copy process, the copy rate drops to 0, in this moment the access point isn't reachable anymore via ping. If I'm lucky, like in this picture, after a short period of time, it goes slowly up again. But I also had it happen too now very often, that the ping breaks down into "internal network error" and the copy process is abrupted. The wifi driver mostly crashes at this moment, and Windows tries to reload the driver.

 

I tried two access points, and it happens on both of them, so the issues seem to lay in the Intel card or its drivers. I am using the latest available drivers for the 9260 card and Windows 10 64bit, version 20.60.0.

 

Receiving files work without an issue, from the LAN to the 9260 card. This issue just happens, when I send away files from the laptop, into the LAN.

 

The connection is 866Mbit (2x80Mhz). I tried two separate access points, both show the same behavior.

 

Edit:

I just noticed that I can randomly crash the Access Point by this (after some tries, copying a large file over the LAN, the access point would restart randomly)

d77mbIh.png

I can see this behavior on two different routers/access points. One is a Linksys WRT1200, another a totally different model and brand by a totally different manufacturer (AVM FRITZ!WLAN Repeater DVB-C).

Re: 82574l gigabit adapter limited to 100Mbps

$
0
0

Hello Saravana_CSB,

Thank you for posting in Intel Wired Ethernet Communities.  This issue involves an embedded Ethernet controller.  Have you checked with Intel Embedded Community on this issue?  Please ask in Intel Embedded Community for assistance.  Please let us know if you have any other questions.

Best regards,
Daniel D
Intel Customer Support
Agent under contract to Intel

Re: XL710 QDA2 link not detected.

$
0
0

Hello Prakashkurup7,

We would like to help you with this issue.  Please inform us if you have not resolved the issue.  If you have any questions please let us know..

Best Regards,
Daniel D
Intel Customer Support
Agent under contract to Intel

Re: "Present Screen" video issue on hub.

$
0
0

Hello,

 

i installed latest Intel Unite app version and issue still exist.

I am looking forward for any help.

 

Krzbes

 

 

Computer Profile Summary

Computer Name:XPR20611 (in WORKGROUP)
Profile Date:poniedziałek, 23 lipiec 2018 12:15:14
Advisor Version:8.6
Windows Logon:Administrator
Operating System
Windows 10 Professional (x64) Version 1803 (build 17134.167)
Install Language: Polski (Polska)
System Locale: Polski (Polska)
Installed: 23.07.2018 12:06:25
Servicing Branch: Current Branch (CB)
Boot Mode: UEFI with Secure Bootdisabled
System Model
Asset Tag: Default string
Chassis Serial Number: Default string
Enclosure Type: Desktop
Processor a
3,40 gigahertz Intel Core i5-7500
256 kilobyte primary memory cache
1024 kilobyte secondary memory cache
6144 kilobyte tertiary memory cache
64-bit ready
Multi-core (4 total)
Not hyper-threaded
Main Circuit Board b
Board: ASUSTeK COMPUTER INC. PRIME H270-PRO Rev X.0x
Serial Number: 171115516100814
Bus Clock: 100 megahertz
UEFI: American Megatrends Inc. 1205 05/03/2018
Drives
499,46 Gigabytes Usable Hard Drive Capacity
472,76 Gigabytes Hard Drive Free Space

 

Samsung SSD 860 EVO 500GB [Hard drive] (500,11 GB) -- drive 0, s/n S3Z2NB0K374466A, rev RVT01B6Q, SMART Status: Healthy
Memory Modules c,d
16328 Megabytes Usable Installed Memory

 

Slot 'ChannelA-DIMM1' is Empty
Slot 'ChannelA-DIMM2' has 8192 MB (serial number 1C33ACFC)
Slot 'ChannelB-DIMM1' is Empty
Slot 'ChannelB-DIMM2' has 8192 MB (serial number 1C33AE99)
Local Drive Volumes
c: (NTFS on drive 0)499,46 GB472,76 GB free
Network Drives
None detected
Users (mouse over user name for details)
local user accountslast logon
DisabledAdministrator23.07.2018 12:06:26(admin)
local system accounts
DisabledGośćnever
DisabledKonto domyślnenever
DisabledWDAGUtilityAccountnever
Font Driver Host domain logons
UMFD-023.07.2018 12:06:16
UMFD-123.07.2018 12:06:16
Disabled Marks a disabled account;Locked out Marks a locked account
Printers
Microsoft Print To PDFon PORTPROMPT:
Microsoft Shared Fax Driveron SHRFAX:
Microsoft XPS Document Writer v4on PORTPROMPT:
Controllers
Intel(R) 300 Series Chipset Family SATA AHCI Controller
Display
NVIDIA GeForce GTX 1050 [Display adapter]
BenQ GL2250H [Monitor] (21,7"vis, s/n N1G00734019, styczeń 2016)
Bus Adapters
Kontroler miejsc do magazynowania firmy Microsoft
ASMedia USB 3.1 eXtensible Host Controller - 1.10 (Microsoft)
Intel(R) USB 3.0 eXtensible Host Controller - 1.0 (Microsoft)
Multimedia
NVIDIA High Definition Audio
Realtek High Definition Audio
Virus Protection [Back to Top]
Windows Defender Version 4.18.1806.18062
    Scan Engine Version 1.1.15100.1
    Virus Definitions Version 23.07.2018 Rev 1.273.202.0
    Realtime File Scanning On
ESET NOD32 Antivirus Version 11.2.49.0
    Virus Definitions Version Up To Date
Group Policies
None detected
Communications
↑ Intel(R) Ethernet Connection (2) I219-V
primaryAuto IP Address:10.10.11.118 / 23
Gateway:10.10.10.1
Dhcp Server:10.10.10.1
Physical Address:18:31:BF:09:23:DF
Connection Speed:1 Gbps
Microsoft Kernel Debug Network Adapter
Networking Dns Server:10.10.10.1
Other Devices
BenQ GL2250H-4 (NVIDIA High Definition Audio)
Realtek Digital Output (Realtek High Definition Audio)
Systemowe oprogramowanie układowe
Urządzenie wejściowe USB (2x)
Urządzenie klawiatury HID [Keyboard]
Mysz zgodna z HID [Mouse]
Fax
Główna kolejka wydruku
Microsoft Print to PDF
Microsoft XPS Document Writer
Microsoft Radio Device Enumeration Bus
Syntezator SW Microsoft GS Wavetable
Główny koncentrator USB (USB 3.0) (2x)
USB Storage Use in past 30 Days (mouse over last used for details)[Back to Top]
None detected
Hosted Virtual Machines (mouse over name for details)[Back to Top]
None detected
Network Map (mouse over IP address for physical address)[Back to Top]
IPDevice TypeDevice DetailsDevice Roles
10.10.11.9Physical Address C8:F4:39:06:0B:27
10.10.11.25WindowsVmatrix1 (in VC)IIS Web Server, Samba Server, Domain Controller
10.10.11.90Vctest (in TEST), MS Hyper-VSamba Server, Domain Controller
10.10.11.97Physical Address C8:F4:39:06:0B:5B
10.10.11.103WindowsVmatrix01 (in VC01)IIS Web Server, Samba Server, Domain Controller
10.10.11.107WindowsVmatrix001 (in VC001)IIS Web Server, Samba Server, Domain Controller
10.10.11.108WindowsVmatrix002 (in VC002)IIS Web Server, Samba Server, Domain Controller
10.10.11.118Windows 10 WorkstationXpr20611 (in WORKGROUP)
10.10.11.138Samsung
Missing Security Updates – for Adobe, Apple, Java, Microsoft and more[Back to Top]

Hotfixes from Windows Update (agent version 10.0.17134.137) install automatically.
Last install: 23.07.2018 11:55:55, check: 23.07.2018 12:14:00.

 

All required security updates have been installed  (using Advisor definitions version 2018.7.13.2) according to the 07/10/2018 Microsoft Security Bulletin Summary and bulletins from other vendors.
Software Licenses [Back to Top]
Belarc - Advisord5857d68
Microsoft - PowerShell89383-100-0001260-04309
Microsoft - Windows 10 Professional (x64)00330-80000-00000-AA222 (Key: none activated)
Software Versions & Usage (mouse over i for details, click i for location)[Back to Top]
     i  Adobe Systems - Shockwave Flash version 30,0,0,134
     i  Adobe Systems - Shockwave Flash version 30,0,0,134 (64-bit)
   ı i  Belarc - Advisor version 8.6
   ı i  ESET - Security version 11.2.49.0 (64-bit)
   ı i  Intel - Delayed launcher version 1, 0, 0, 1
   ı i  Intel - IAStorDataSvc version 16.0.2.1086
     i  Intel - IAStorUI version 16.0.2.1086
     i  Microsoft - .NET Framework version 4.7.3056.0
     i  Microsoft - 1527c705-839a-4832-9118-54d4Bd6a0c89 version 10.0.17134.1
     i  Microsoft - Antimalware Service Executable version 4.18.1806.18062 (64-bit)
     i  Microsoft - Aplikacja przypisanego dostępu na ekran blokady version 1000.17134.1.0
     i  Microsoft - App Installer version 1.0.12271.0 (64-bit)
     i  Microsoft - AsyncTextService version 10.0.17134.1
     i  Microsoft - c5e2524a-ea46-4f67-841f-6a9465d9d515 version 10.0.17134.1
     i  Microsoft - CameraBarcodeScannerPreview version 10.0.17134.1
     i  Microsoft - CapturePicker version 10.0.17134.1
     i  Microsoft - Cortana version 1.10.7.17134
     i  Microsoft - Credential Dialog version 10.0.17134.1
     i  Microsoft - Domyślny ekran blokady systemu Windows version 10.0.17134.1
     i  Microsoft - E2A4F912-2574-4A75-9BB0-0D023378592B version 10.0.17134.1
     i  Microsoft - Edge version 42.17134.1.0
     i  Microsoft - Eye Control version 10.0.17134.1
     i  Microsoft - Feedback Hub version 1.1712.612.0 (64-bit)
     i  Microsoft - Funkcje rodzinne firmy Microsoft version 1000.17134.1.0
     i  Microsoft - Get Help version 10.1706.10441.0 (64-bit)
     i  Microsoft - Groove Music version 10.17112.19011.0 (64-bit)
     i  Microsoft - Instalator usługi Windows Hello version 10.0.17134.1
   ı i  Microsoft - Instalator Windows — Unicode version 5.0.17134.1
   ı i  Microsoft - Instalator Windows — Unicode version 5.0.17134.1 (64-bit)
     i  Microsoft - Internet Explorer version 11.00.17134.1
     i  Microsoft - Internet Explorer version 11.00.17134.1 (64-bit)
     i  Microsoft - Klient programu Microsoft Edge DevTools version 1000.17134.1.0
     i  Microsoft - Mail and Calendar version 17.8827.22055.0 (64-bit)
     i  Microsoft - Messaging version 3.38.22001.0 (64-bit)
     i  Microsoft - Microsoft.Microsoft3DViewer version 2.1803.8022.0 (64-bit)
     i  Microsoft - Mixed Reality Portal version 10.0.17134.1
     i  Microsoft - Mobile Plans version 4.1801.521.0 (64-bit)
     i  Microsoft - Movies & TV version 10.17112.19011.0 (64-bit)
     i  Microsoft - MSN Weather version 4.22.3254.0 (64-bit)
     i  Microsoft - My Office version 17.8918.5926.0 (64-bit)
     i  Microsoft - Okno dialogowe Dodawanie sugestii folderów version 10.0.17134.1
   ı i  Microsoft - OneDrive version 18.111.0603.0006
     i  Microsoft - OneNote version 17.8827.20991.0 (64-bit)
     i  Microsoft - Paint 3D version 3.1803.5027.0 (64-bit)
     i  Microsoft - Pay version 2.1.18009.0 (64-bit)
     i  Microsoft - People version 10.3.10452.0 (64-bit)
     i  Microsoft - Photos version 2018.18011.15918.0 (64-bit)
     i  Microsoft - PinningConfirmationDialog version 1000.17134.1.0
     i  Microsoft - Poczta e-mail i konta version 10.0.17134.1
     i  Microsoft - Połącz version 10.0.17134.1
     i  Microsoft - Print 3D version 2.0.3621.0 (64-bit)
     i  Microsoft - PrintDialog version 6.2.0.0
     i  Microsoft - Przepływ portalu uwierzytelniania version 10.0.17134.1
     i  Microsoft - Przepływ połączenia sieciowego version 10.0.17134.1
     i  Microsoft - Shell Input Application version 1000.17134.1.0
     i  Microsoft - Sticky Notes version 2.0.13.0 (64-bit)
     i  Microsoft - Store version 11712.1001.23.0 (64-bit)
     i  Microsoft - Store Experience Host version 11712.1801.10002.0 (64-bit)
     i  Microsoft - Tips version 6.9.10602.0 (64-bit)
     i  Microsoft - Ustawienia version 10.0.2.1000
     i  Microsoft - Web Media Extensions version 1.0.3102.0 (64-bit)
     i  Microsoft - Win32WebViewHost version 10.0.17134.1
     i  Microsoft - Windows Alarms & Clock version 10.1712.10611.0 (64-bit)
     i  Microsoft - Windows Calculator version 10.1712.10601.0 (64-bit)
     i  Microsoft - Windows Camera version 2017.1117.80.0 (64-bit)
     i  Microsoft - Windows Defender Security Center version 10.0.17134.1
     i  Microsoft - Windows Defender SmartScreen version 1000.17134.1.0
     i  Microsoft - Windows Live version 17.3.6816.0313
     i  Microsoft - Windows Maps version 5.1711.10401.0 (64-bit)
     i  Microsoft - Windows Media Player version 12.0.17134.1
     i  Microsoft - Windows Media Player version 12.0.17134.1 (64-bit)
     i  Microsoft - Windows Script Host version 5.812.10240.16384
     i  Microsoft - Windows Script Host version 5.812.10240.16384 (64-bit)
     i  Microsoft - Windows Voice Recorder version 10.1712.10611.0 (64-bit)
     i  Microsoft - Wypełnij test version 10.0.17134.1
   ı i  Microsoft - Wyszukiwanie Windows version 7.0.17134.1
   ı i  Microsoft - Wyszukiwanie Windows version 7.0.17134.1 (64-bit)
     i  Microsoft - Xbox version 38.38.14002.0 (64-bit)
     i  Microsoft - Xbox Game bar version 1.26.6001.0 (64-bit)
     i  Microsoft - Xbox Game Speech Window version 1.17.29001.0 (64-bit)
     i  Microsoft - Xbox Game UI version 1000.17134.1.0
     i  Microsoft - Xbox gaming overlay version 1.15.1001.0 (64-bit)
     i  Microsoft - Xbox Identity Provider version 12.36.15002.0 (64-bit)
     i  Microsoft - Xbox TCUI version 1.11.28003.0 (64-bit)
     i  Microsoft - Zawartość firmy Microsoft version 10.0.17134.1
     i  Microsoft - Środowisko powłoki systemu Windows version 10.0.17134.1
     i  Microsoft Studios - Solitaire Collection version 4.0.1301.0
     i  No Company Name - OpenSSH for Windows OpenSSH_7.6p1 for Windows (64-bit)
     i  NVIDIA - 3D Vision Photo Viewer version 7.17.13.8813
   ı i  NVIDIA - Container gs_03_40 0 (64-bit)
   ı i  NVIDIA - GeForce 3D Vision version 7.17.13.8813
     i  Outlook - Twoje konto version 10.0.17134.1
     i  Przypisane przez organizację - Konto służbowe version 1000.17134.1.0
   ı i  Realtek Semiconductor - Menedżer Realtek HD Audio version 1.0.561.0 (64-bit)
     i  Skype -  version 12.13.274.0 (64-bit)

Predictive Failure on power Supply

$
0
0

Hi,

 

I'm using a S2600CP server board. The integrated BMW Web Console is reporting power supply warnings on a monthly base.

 

The server has 2 power supplies, one connected to the net the other one is connected to a ups.

Both power suppliers are generating the same warning (at the same time):

Event that generated this alert:

RID:02B2 TS:07/21/2018 09:01:02 SN:PS2 Status ST:Power Supply ED:Predictive Failure ET:Asserted EC:Non-Critical

RID:02B2 RT:02 TS:5B52F64E GID:0020 ER:04 ST:08 S#:51 ET:6F ED:A2 06 30 EX:01 FF FF FF FF FF FF FF

Event that generated this alert:

RID:02B1 TS:07/21/2018 09:01:02 SN:PS1 Status ST:Power Supply ED:Predictive Failure ET:Asserted EC:Non-Critical

RID:02B1 RT:02 TS:5B52F64E GID:0020 ER:04 ST:08 S#:50 ET:6F ED:A2 06 30 EX:01 FF FF FF FF FF FF FF

 

The BMC warning indication keeps blinking and the Server Health reports a warning:

Pwr Unit Redundreports full redundancy has been regained Warning0x0001

 

The BMC version:

BMC FW Rev : 01.21.6038

 

The event generated looks like a false event, but i'm concerned about the Pwr Unit Redund warning.

What can cause this warning?

 

Thanks in advance,

Davy


Re: Additional datasets request

$
0
0

Hi
Could you please let us know if the above solution helped for you?
Thanks

Re: how to install pytorch in a new conda env or steps to use the default one

$
0
0

Hello,

As we haven't received any response we are closing this ticket. Feel free to open a new thread in case you face any further issues.

Re: What are the GPU specifications provided in 1 month AI-cloud colfax research program?

$
0
0

Hi ,
As we haven't received any response we are closing this thread.
If you have any other issue,Please raise a new thread
Thanks
 

Re: Job crashes, error file is empty

$
0
0

This wasn't the case.

As I mentioned before, the job, when exeucted on Jupyetr terminal, terminates after few seconds (way before the time limit of the session).

 

Also,

when executing the jobe as we discussed yestereday,

I obtained a quota execceded error,

which I dont get when using the jupyter notebook/terminal.

Jupyter hub crashes when running multiprocessing (6cores) script

$
0
0

Hi, I'm running a very simple script,

however the entire system crashes after 2-3 minutes of running.

 

import os

import json

import pandas

import sys

import multiprocessing as mp

 

 

def jsons2df(path):

    json_list=[]

    file_list = os.scandir(path)

    counter=0

    error_counter=0

    for j_file in file_list:

        counter+=1

        if counter%10000==0:

            print(counter)

        #with open(os.path.join(path,j_file),"r") as j:

        with open(j_file.path,"r") as j:

            try:

                data = json.load(j)

            except:

                print("ERROR: ",j_file)

                error_counter+=1

                if error_counter>1000:

                    break

                else:

                    continue

            json_list.append(data)

    try:

        json_df= pandas.DataFrame(json_list)

        b=json_df["url"].apply(lambda x: x.split("/")[2])

        json_df['domain']=b

        return json_df

    except:

        return json_list

   

def get_and_extract(number):

    print("starting: ",number)

    if not os.path.isfile("%s.tar"%(number)):

        if not os.path.isdir("%s"%(number)):

            if not os.path.isfile("./pickles/wiki_%s.pickle"%(number)):

                print("downloading")

                url="http://data.dws.informatik.uni-mannheim.de/webtables/2015-07/englishCorpus/compressed/%s.tar.gz"%(number)

                os.system("wget %s"%(url))

                print("extracting gz #1")

                os.system("tar -xzf %s.tar.gz >/dev/null"%(number))

                os.system("rm -rf %s.tar.gz"%(number))

 

 

    if  os.path.isfile("%s.tar"%(number)):

        print("extracting tar #2")

        os.system("tar --skip-old-files -xf %s.tar >/dev/null"%(number))

        os.system("rm -rf %s.tar"%(number))

 

 

    if not os.path.isfile("%s.tar"%(number)):

        if number.startswith("0"):

            number=number[1:]

        path="./%s"%(number)

        if not os.path.isfile("./pickles/wiki_%s.pickle"%(number)):

            df =jsons2df(path)

            print("pickling:")

            df.to_pickle("./pickles/df_%s.pickle"%(number))

            b=df[df.domain.str.contains("en.wikipedia.org")]

            b=b[~b.url.str.contains('Special:Book')]

            b.to_pickle("./pickles/wiki_%s.pickle"%(number))

        if os.path.isdir("%s"%(number)):

            print("Deleting:")

            os.system("rsync -r --delete emptydir/ %s/"%(number))

            os.system("rmdir %s"%(number))

            print("Done: ",number)   

#get_and_extract(number)

 

 

def main(number1,number2):

         

    #pool=mp.Pool(CORES)

    pool = mp.Pool(6)

    print(number1,number2)

    print("Looping:")

    for i in range(int(number1),int(number2)+1):

        print(i)

        x =str(i)

        y=0

        x =str(i)

        y=0

        a=pool.apply_async(get_and_extract,(x,))

        #a=pool.apply_async(get_and_extract,(x,))

    #print(a)

    pool.close()

"""

if __name__ == '__main__':

    number=sys.argv[1]

    number2=sys.argv[2]

    main(number,number2)

"""

Viewing all 18600 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>